Sep 18 16:02:50.615980 ip-10-0-130-195 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 16:02:50.615994 ip-10-0-130-195 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 16:02:50.616003 ip-10-0-130-195 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 16:02:50.616466 ip-10-0-130-195 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 16:03:00.659925 ip-10-0-130-195 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 16:03:00.659942 ip-10-0-130-195 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 54763fb502864f80962ad999d352dbb1 --
Sep 18 16:05:03.577846 ip-10-0-130-195 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 16:05:07.073083 ip-10-0-130-195 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:05:07.073083 ip-10-0-130-195 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 16:05:07.073083 ip-10-0-130-195 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:05:07.073083 ip-10-0-130-195 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 16:05:07.073083 ip-10-0-130-195 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:05:07.073781 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.073154    2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 16:05:07.076743 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076727    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:05:07.076743 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076743    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076748    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076751    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076754    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076757    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076760    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076763    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076765    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076768    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076771    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076773    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076777    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076780    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076782    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076785    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076789    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076793    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076796    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076799    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:05:07.076804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076803    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076806    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076809    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076812    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076815    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076818    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076821    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076824    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076826    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076829    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076831    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076834    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076836    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076839    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076842    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076846    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076848    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076850    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076853    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076856    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:05:07.077257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076859    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076861    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076864    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076866    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076869    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076871    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076874    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076876    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076878    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076881    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076883    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076886    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076888    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076891    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076894    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076896    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076899    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076902    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076904    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:05:07.077760 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076908    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076912    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076915    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076917    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076920    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076923    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076926    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076928    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076931    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076933    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076936    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076938    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076941    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076944    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076946    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076949    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076951    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076954    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076956    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076959    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076961    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:05:07.078267 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076964    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076966    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076969    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076971    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076974    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076976    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076980    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076982    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076985    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076987    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076990    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076992    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076995    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.076997    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077000    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077002    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077004    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077007    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077009    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077742    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:05:07.078807 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077749    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077752    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077755    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077758    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077762    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077766    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077769    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077773    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077776    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077778    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077783    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077786    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077789    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077792    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077795    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077798    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077800    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077803    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077806    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077808    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:05:07.079327 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077811    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077814    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077817    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077820    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077823    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077825    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077828    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077831    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077833    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077836    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077838    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077841    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077843    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077846    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077848    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077851    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077853    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077856    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077858    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:05:07.079852 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077861    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077865    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077868    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077871    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077874    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077876    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077878    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077881    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077884    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077888    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077891    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077895    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077897    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077900    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077902    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077905    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077908    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077910    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077913    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:05:07.080321 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077915    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077918    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077920    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077922    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077925    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077927    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077930    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077932    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077935    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077937    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077940    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077942    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077944    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077948    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077950    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077953    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077955    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077958    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077961    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077964    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:05:07.080790 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077966    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077969    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077971    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077974    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077976    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077979    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077981    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077984    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077986    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077989    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077991    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077994    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077996    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.077999    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078001    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078004    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078006    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078009    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078011    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078014    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:05:07.081270 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078379    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078383    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078386    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078389    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078392    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078396    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078398    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078401    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078404    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078406    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078408    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078411    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078414    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078416    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078419    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078421    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078424    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078427    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078430    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078432    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:05:07.081774 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078435    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078438    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078442    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078446    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078448    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078451    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078454    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078457    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078460    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078463    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078465    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078468    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078471    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078474    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078476    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078479    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078481    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078484    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078488    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078490    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:05:07.082258 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078493    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078495    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078498    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078500    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078502    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078505    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078507    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078509    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078516    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078534    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078537    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078539    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078542    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078545    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078547    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078551    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078553    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078556    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078558    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078561    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:05:07.082804 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078563    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078566    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078568    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078571    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078573    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078576    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078578    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078582    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078585    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078590    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078592    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078595    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078600    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078603    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078606    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078610    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078612    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078615    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078618    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:05:07.083294 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078620    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078624    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078627    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078629    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078632    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078634    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078637    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078640    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078642    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078644    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078647    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078650    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078652    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078654    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078657    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078659    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078661    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078664    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078666    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.078669    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:05:07.083793 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078738    2554 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078778    2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078789    2554 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078794    2554 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078797    2554 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078801    2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078805    2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078809    2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078812    2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078816    2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078819    2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078823    2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078826    2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078830    2554 flags.go:64] FLAG: --cgroup-root=""
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078832    2554 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078835    2554 flags.go:64] FLAG: --client-ca-file=""
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078838    2554 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078841    2554 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078845    2554 flags.go:64] FLAG: --cluster-domain=""
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078847    2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078850    2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078853    2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078857    2554 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078860    2554 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 16:05:07.084291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078863    2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078866    2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078869    2554 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078872    2554 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078875    2554 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078878    2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078881    2554 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078884    2554 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078888    2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078891    2554 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078895    2554 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078898    2554 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078901    2554 flags.go:64] FLAG: --enable-server="true"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078904    2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078909    2554 flags.go:64] FLAG: --event-burst="100"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078912    2554 flags.go:64] FLAG: --event-qps="50"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078915    2554 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078918    2554 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078921    2554 flags.go:64] FLAG: --eviction-hard=""
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078924    2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078927    2554 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078932    2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078935    2554 flags.go:64] FLAG: --eviction-soft=""
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078938    2554 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078941    2554 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 16:05:07.084881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078944    2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078946    2554 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078949    2554 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078952    2554 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078954    2554 flags.go:64] FLAG: --feature-gates=""
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078958    2554 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078961    2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078964    2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078967    2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078970    2554 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078973    2554 flags.go:64] FLAG: --help="false"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078976    2554 flags.go:64] FLAG: --hostname-override="ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078979    2554 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078982    2554 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078984    2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078988    2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078991    2554 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078995    2554 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.078998    2554 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079001    2554 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079004    2554 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079006    2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079010    2554 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079013    2554 flags.go:64] FLAG: --kube-reserved=""
Sep 18 16:05:07.085507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079015    2554 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079018    2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079021    2554 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079023    2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079027    2554 flags.go:64] FLAG: --lock-file=""
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079030    2554 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079033    2554 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079036    2554 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079041    2554 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079043    2554 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079046    2554 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079049    2554 flags.go:64] FLAG: --logging-format="text"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079052    2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079055    2554 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079057    2554 flags.go:64] FLAG: --manifest-url=""
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079060    2554 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079065    2554 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079068    2554 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079072    2554 flags.go:64] FLAG: --max-pods="110"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079074    2554 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079077    2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079080    2554 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079083    2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079086    2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079089    2554 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 16:05:07.086225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079093    2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079101    2554 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079104    2554 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079107    2554 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079110    2554 flags.go:64] FLAG: --pod-cidr=""
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079112    2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079118    2554 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079121    2554 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079124    2554 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079127    2554 flags.go:64] FLAG: --port="10250"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079130    2554 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079132    2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bdc0289a0cd9b308"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079137    2554 flags.go:64] FLAG: --qos-reserved=""
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079139    2554 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079142    2554 flags.go:64] FLAG: --register-node="true"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079145    2554 flags.go:64] FLAG: --register-with-taints=""
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079148    2554 flags.go:64] FLAG: --registry-burst="10"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079151    2554 flags.go:64] FLAG: --registry-qps="5"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079154    2554 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079157    2554 flags.go:64] FLAG: --reserved-memory=""
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079160    2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079163    2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079166    2554 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079169    2554 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079172    2554 flags.go:64] FLAG: --runonce="false"
Sep 18 16:05:07.087171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079174    2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079177    2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079180    2554 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079183    2554 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079185    2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079188    2554 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079191    2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079195    2554 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079198    2554 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079201    2554 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079204    2554 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079207    2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079209    2554 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079212    2554 flags.go:64] FLAG: --system-cgroups=""
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079215    2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079220    2554 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079224    2554 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079226    2554 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079230    2554 flags.go:64] FLAG: --tls-min-version=""
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079233    2554 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079236    2554 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079239    2554 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079242    2554 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079244    2554 flags.go:64] FLAG: --v="2"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079248    2554 flags.go:64] FLAG: --version="false"
Sep 18 16:05:07.087975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079252    2554 flags.go:64] FLAG: --vmodule=""
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079256    2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.079259    2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079388    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079392    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079396    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079400    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079404    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079408    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079412    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079416    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079418    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079421    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079423    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079427    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079430    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079432    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079435    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079437    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079440    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:05:07.088693 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079442    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079445    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079449    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079451    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079454    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079457    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079460    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079462    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079466    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079470    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079473    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079476    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079478    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079481    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079484    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079487    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079490    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079492    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079495    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:05:07.089254 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079497    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079500    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079502    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079505    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079507    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079509    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079513    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079517    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079535    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079537    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079540    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079542    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079545    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079548    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079550    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079553    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079558    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079561    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079564    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:05:07.089824 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079567    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079569    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079572    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079575    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079577    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079580    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079583    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079585    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079588    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079591    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079593    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079596    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079598    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079601    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079603    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079606    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079608    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079611    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079613    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079616    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:05:07.090345 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079618    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079621    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079623    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079626    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079628    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079631    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079634    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079636    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079639    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079641    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079644    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079647    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079658    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079661    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079664    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079666    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079669    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079672    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079674    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079678    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:05:07.090948 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079681    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:05:07.091508 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079684    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:05:07.091508 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079687    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:05:07.091508 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.079690    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:05:07.091508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.080291    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:05:07.091508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.088773    2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 16:05:07.091508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.088789    2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 16:05:07.091508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.088805    2554 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 16:05:07.091508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.088811    2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 16:05:07.091508 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088866    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:05:07.091508 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088871    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:05:07.091508 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088873    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:05:07.091508 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088876    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:05:07.091508 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088879    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:05:07.091508 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088882    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:05:07.091508 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088885    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088889    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088893    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088895    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088898    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088901    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088903    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088905    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088908    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088910    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088913    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088916    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088918    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088921    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088923    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088925    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088928    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088930    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088934    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088936    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:05:07.091946 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088940    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088942    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088945    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088947    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088950    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088952    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088955    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088957    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088960    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088962    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088965    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088968    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088971    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088973    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088975    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088978    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088980    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088983    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088985    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088988    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:05:07.092478 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088990    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088993    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088997    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.088999    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089002    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089004    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089007    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089009    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089011    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089014    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089017    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089019    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089022    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089024    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089027    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089029    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089032    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089034    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089037    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089039    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:05:07.092991 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089042    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089044    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089046    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089049    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089051    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089054    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089057    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089059    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089062    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089064    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089066    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089069    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089072    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089075    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089078    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089080    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089084    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089088    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089091    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089094    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:05:07.093535 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089096    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:05:07.094033 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089099    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:05:07.094033 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089101    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:05:07.094033 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089104    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:05:07.094033 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089106    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:05:07.094033 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089109    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:05:07.094033 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089112    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:05:07.094033 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089115    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:05:07.094033 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089118    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:05:07.094033 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089121    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:05:07.094033 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089124    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:05:07.094033 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089126    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:05:07.094033 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089129    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:05:07.094033 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.089133    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:05:07.094033 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089244    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089249    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089252    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089255    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089258    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089261    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089264    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089267    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089270    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089272    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089275    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089277    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089280    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089283    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089286    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089288    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089291    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089294    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089296    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089298    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:05:07.094376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089301    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089304    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089306    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089308    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089311    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089313    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089316    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089318    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089320    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089323    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089325    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089328    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089331    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089333    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089336    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089338    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089341    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089343    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089346    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089348    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:05:07.094915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089351    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089355    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089357    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089360    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089362    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089366    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089369    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089371    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089374    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089376    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089379    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089381    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089384    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089387    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089389    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089392    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089394    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089396    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089399    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:05:07.095401 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089401    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089405    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089408    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089411    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089414    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089417    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089420    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089422    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089425    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089427    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089430    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089432    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089435    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089437    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089440    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089443    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089447    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089450    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089453    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:05:07.095891 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089456    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089459    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089462    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089464    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089467    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089469    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089472    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089475    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089477    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089480    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089482    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089485    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089487    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089490    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089492    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089495    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089497    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089500    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089502    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089505    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:05:07.096342 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:07.089507    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:05:07.096867 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.089512    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:05:07.096867 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.089628    2554 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 16:05:07.096867 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.091593    2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 16:05:07.096867 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.092646    2554 server.go:1019] "Starting client certificate rotation"
Sep 18 16:05:07.096867 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.092742    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:05:07.096867 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.093229    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:05:07.096867 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.094964    2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:05:07.096867 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.096170    2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:05:07.100381 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.100366    2554 log.go:25] "Validated CRI v1 runtime API"
Sep 18 16:05:07.103378 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.103364    2554 log.go:25] "Validated CRI v1 image API"
Sep 18 16:05:07.104225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.104208    2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 16:05:07.107137 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.107116    2554 fs.go:135] Filesystem UUIDs: map[52a6e277-534d-48f1-b329-f903bfa5a445:/dev/nvme0n1p4 552c6d30-89ca-46ee-bca3-8c4329bac7a8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 18 16:05:07.107220 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.107136    2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 16:05:07.111468 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.111363    2554 manager.go:216] Machine: {Timestamp:2026-09-18 16:05:07.110835044 +0000 UTC m=+2.058333808 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100140 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec226cd1db92eaaf15c444823e02fabb SystemUUID:ec226cd1-db92-eaaf-15c4-44823e02fabb BootID:54763fb5-0286-4f80-962a-d999d352dbb1 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c0:1a:c1:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c0:1a:c1:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:46:a6:5a:c0:93 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 16:05:07.111468 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.111465    2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 16:05:07.111591 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.111565    2554 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 16:05:07.112918 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.112891    2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 16:05:07.113054 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.112921    2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-195.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 16:05:07.114011 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.114001    2554 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 16:05:07.114043 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.114016    2554 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 16:05:07.114043 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.114024    2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:05:07.114043 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.114034    2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:05:07.114125 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.114103    2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 16:05:07.117488 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.117478    2554 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:05:07.117611 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.117601    2554 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 16:05:07.117641 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.117634    2554 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 16:05:07.117668 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.117642    2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 16:05:07.117668 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.117655    2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 16:05:07.117668 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.117664    2554 kubelet.go:398] "Adding apiserver pod source"
Sep 18 16:05:07.117746 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.117672    2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 16:05:07.118580 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.118558    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:05:07.118702 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.118589    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:05:07.121655 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.121628    2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 16:05:07.124585 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.124566    2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 16:05:07.124663 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.124611    2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 16:05:07.125513 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.125501    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 16:05:07.125570 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.125536    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 16:05:07.125570 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.125547    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 16:05:07.125570 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.125555    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 16:05:07.125570 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.125564    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 16:05:07.125570 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.125570    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 16:05:07.125711 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.125584    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 16:05:07.125711 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.125589    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 16:05:07.125711 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.125596    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 16:05:07.125711 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.125601    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 16:05:07.125711 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.125609    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 16:05:07.125711 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.125614    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 16:05:07.125711 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.125641    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 16:05:07.125711 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.125646    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 16:05:07.126875 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.126856    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:05:07.129747 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.129725    2554 server.go:1268] "Started kubelet"
Sep 18 16:05:07.129824 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.129791    2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 16:05:07.129956 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.129929    2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 16:05:07.130056 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.129966    2554 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 16:05:07.130659 ip-10-0-130-195 systemd[1]: Started Kubernetes Kubelet.
Sep 18 16:05:07.131133 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.130988    2554 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 16:05:07.131483 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.131416    2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 16:05:07.132244 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.132215    2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-195.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:05:07.132830 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.132808    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:05:07.132886 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.132811    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-195.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:05:07.134759 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.134739    2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 16:05:07.136778 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.136761    2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 16:05:07.136861 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.136783    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 16:05:07.136861 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.136843    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-195.ec2.internal\" not found"
Sep 18 16:05:07.137256 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.137240    2554 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 16:05:07.137256 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.137257    2554 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 16:05:07.137394 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.137244    2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 16:05:07.137448 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.137415    2554 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 16:05:07.137448 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.137424    2554 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 16:05:07.137838 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.137823    2554 factory.go:55] Registering systemd factory
Sep 18 16:05:07.137909 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.137841    2554 factory.go:223] Registration of the systemd container factory successfully
Sep 18 16:05:07.138079 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.138061    2554 factory.go:153] Registering CRI-O factory
Sep 18 16:05:07.138079 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.138080    2554 factory.go:223] Registration of the crio container factory successfully
Sep 18 16:05:07.138238 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.138130    2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 16:05:07.138238 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.138149    2554 factory.go:103] Registering Raw factory
Sep 18 16:05:07.138238 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.138164    2554 manager.go:1196] Started watching for new ooms in manager
Sep 18 16:05:07.138605 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.138590    2554 manager.go:349] Starting recovery of all containers
Sep 18 16:05:07.138783 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.138762    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:05:07.138955 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.138919    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-195.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 16:05:07.161575 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.161516    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6k9vn"
Sep 18 16:05:07.164343 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.164318    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6k9vn"
Sep 18 16:05:07.167314 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.167288    2554 manager.go:354] Recovery completed
Sep 18 16:05:07.171413 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.171398    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:05:07.173722 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.173708    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:05:07.173770 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.173738    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:05:07.173770 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.173748    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:05:07.174329 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.174318    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:05:07.174381 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.174319    2554 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 16:05:07.174381 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.174350    2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 16:05:07.174381 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.174365    2554 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:05:07.175046 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.175024    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:05:07.175106 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.175051    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:05:07.175106 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.175063    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:05:07.176573 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.176561    2554 policy_none.go:49] "None policy: Start"
Sep 18 16:05:07.176610 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.176578    2554 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 16:05:07.176610 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.176588    2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 16:05:07.178546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.178535    2554 policy_none.go:47] "Start"
Sep 18 16:05:07.217265 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.217106    2554 manager.go:337] "Starting Device Plugin manager"
Sep 18 16:05:07.236425 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.217290    2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 16:05:07.236425 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.217301    2554 server.go:87] "Starting device plugin registration server"
Sep 18 16:05:07.236425 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.217584    2554 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 16:05:07.236425 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.217610    2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 16:05:07.236425 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.217671    2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 16:05:07.236425 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.217753    2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 16:05:07.236425 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.217760    2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 16:05:07.236425 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.218378    2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 16:05:07.236425 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.218448    2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-195.ec2.internal\" not found"
Sep 18 16:05:07.269289 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.269256    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 16:05:07.270622 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.270595    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 16:05:07.270719 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.270644    2554 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 16:05:07.270719 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.270670    2554 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 16:05:07.270817 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.270716    2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 16:05:07.272479 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.272462    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:05:07.318705 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.318666    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:05:07.319668 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.319644    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:05:07.319763 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.319684    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:05:07.319763 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.319697    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:05:07.319763 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.319721    2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.327755 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.327710    2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.327755 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.327730    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": node \"ip-10-0-130-195.ec2.internal\" not found"
Sep 18 16:05:07.344939 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.344917    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-195.ec2.internal\" not found"
Sep 18 16:05:07.371113 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.371081    2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-195.ec2.internal"]
Sep 18 16:05:07.371220 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.371157    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:05:07.372642 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.372626    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:05:07.372718 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.372667    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:05:07.372718 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.372681    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:05:07.374946 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.374932    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:05:07.375094 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.375077    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.375134 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.375117    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:05:07.375804 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.375788    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:05:07.375862 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.375822    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:05:07.375862 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.375832    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:05:07.375930 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.375791    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:05:07.375930 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.375895    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:05:07.375930 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.375907    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:05:07.378403 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.378390    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.378444 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.378414    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:05:07.379142 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.379126    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:05:07.379231 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.379154    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:05:07.379231 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.379170    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:05:07.402195 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.402170    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-195.ec2.internal\" not found" node="ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.406712 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.406696    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-195.ec2.internal\" not found" node="ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.439900 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.439873    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1ea19d78c8a13e00dcd379551f4b0152-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal\" (UID: \"1ea19d78c8a13e00dcd379551f4b0152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.440015 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.439922    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ea19d78c8a13e00dcd379551f4b0152-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal\" (UID: \"1ea19d78c8a13e00dcd379551f4b0152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.440015 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.439947    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8694b5494e0c8ea7c82d6e3efeab8b4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-195.ec2.internal\" (UID: \"f8694b5494e0c8ea7c82d6e3efeab8b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.445964 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.445945    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-195.ec2.internal\" not found"
Sep 18 16:05:07.540407 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.540380    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1ea19d78c8a13e00dcd379551f4b0152-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal\" (UID: \"1ea19d78c8a13e00dcd379551f4b0152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.540564 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.540408    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ea19d78c8a13e00dcd379551f4b0152-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal\" (UID: \"1ea19d78c8a13e00dcd379551f4b0152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.540564 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.540426    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8694b5494e0c8ea7c82d6e3efeab8b4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-195.ec2.internal\" (UID: \"f8694b5494e0c8ea7c82d6e3efeab8b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.540564 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.540453    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f8694b5494e0c8ea7c82d6e3efeab8b4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-195.ec2.internal\" (UID: \"f8694b5494e0c8ea7c82d6e3efeab8b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.540564 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.540489    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ea19d78c8a13e00dcd379551f4b0152-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal\" (UID: \"1ea19d78c8a13e00dcd379551f4b0152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.540564 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.540489    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1ea19d78c8a13e00dcd379551f4b0152-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal\" (UID: \"1ea19d78c8a13e00dcd379551f4b0152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.546489 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.546475    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-195.ec2.internal\" not found"
Sep 18 16:05:07.646726 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.646638    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-195.ec2.internal\" not found"
Sep 18 16:05:07.705850 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.705814    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.709672 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:07.709651    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.ec2.internal"
Sep 18 16:05:07.746896 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.746868    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-195.ec2.internal\" not found"
Sep 18 16:05:07.847395 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.847351    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-195.ec2.internal\" not found"
Sep 18 16:05:07.947935 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:07.947858    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-195.ec2.internal\" not found"
Sep 18 16:05:08.048390 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:08.048355    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-195.ec2.internal\" not found"
Sep 18 16:05:08.092615 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:08.092591    2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 16:05:08.093035 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:08.092708    2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 16:05:08.149370 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:08.149338    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-195.ec2.internal\" not found"
Sep 18 16:05:08.170883 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:08.170853    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 16:00:07 +0000 UTC" deadline="2028-05-25 19:52:21.799140071 +0000 UTC"
Sep 18 16:05:08.170883 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:08.170879    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14763h47m13.628264061s"
Sep 18 16:05:08.249633 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:08.249564    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-195.ec2.internal\" not found"
Sep 18 16:05:08.304938 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:08.304913    2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:05:08.321432 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:08.321403    2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:05:08.337052 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:08.337027    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.ec2.internal"
Sep 18 16:05:08.346352 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:08.346335    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:05:08.348837 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:08.348823    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal"
Sep 18 16:05:08.358006 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:08.357986    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:05:08.398726 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:08.398678    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8694b5494e0c8ea7c82d6e3efeab8b4.slice/crio-5c8343d081ba925800ad8e3b97be7365e546fa9ced41ea7eb7f476a188a716a6 WatchSource:0}: Error finding container 5c8343d081ba925800ad8e3b97be7365e546fa9ced41ea7eb7f476a188a716a6: Status 404 returned error can't find the container with id 5c8343d081ba925800ad8e3b97be7365e546fa9ced41ea7eb7f476a188a716a6
Sep 18 16:05:08.404279 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:08.404263    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:05:08.437739 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:08.437715    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea19d78c8a13e00dcd379551f4b0152.slice/crio-25999dfbe78caa8fb58751ddfba7e0c86f6ce7660e2054fa2e59ec5ef981aebb WatchSource:0}: Error finding container 25999dfbe78caa8fb58751ddfba7e0c86f6ce7660e2054fa2e59ec5ef981aebb: Status 404 returned error can't find the container with id 25999dfbe78caa8fb58751ddfba7e0c86f6ce7660e2054fa2e59ec5ef981aebb
Sep 18 16:05:09.098245 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.098163    2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:05:09.118097 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.118064    2554 apiserver.go:52] "Watching apiserver"
Sep 18 16:05:09.122983 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.122959    2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 16:05:09.123343 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.123323    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal","openshift-multus/multus-w26s7","openshift-network-operator/iptables-alerter-f77cv","openshift-ovn-kubernetes/ovnkube-node-f79kb","kube-system/global-pull-secret-syncer-c8d4n","kube-system/konnectivity-agent-vzt5f","kube-system/kube-apiserver-proxy-ip-10-0-130-195.ec2.internal","openshift-cluster-node-tuning-operator/tuned-wwwrj","openshift-image-registry/node-ca-26zfw","openshift-multus/multus-additional-cni-plugins-nw2sh","openshift-multus/network-metrics-daemon-6psxt","openshift-network-diagnostics/network-check-target-j5sx5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj","openshift-dns/node-resolver-9286k"]
Sep 18 16:05:09.126717 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.125929    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.128133 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.128109    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.128791 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.128472    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 16:05:09.128791 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.128535    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 16:05:09.128791 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.128569    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vx59n\""
Sep 18 16:05:09.128791 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.128645    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 16:05:09.128791 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.128674    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 16:05:09.128791 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.128704    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 16:05:09.130075 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.130058    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 16:05:09.130158 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.130112    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 16:05:09.130375 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.130355    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f77cv"
Sep 18 16:05:09.130960 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.130619    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 16:05:09.130960 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.130823    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hvq9z\""
Sep 18 16:05:09.133598 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.132804    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:05:09.133598 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.132867    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.133598 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.132900    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8cs6v\""
Sep 18 16:05:09.133598 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.133129    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 16:05:09.133598 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.133565    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 16:05:09.134834 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.134816    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 16:05:09.135386 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.135365    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 16:05:09.135470 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.135447    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 16:05:09.135557 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.135540    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h4v9d\""
Sep 18 16:05:09.135756 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.135737    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 16:05:09.135868 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.135851    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 16:05:09.135921 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.135665    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 16:05:09.136018 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.136005    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:09.136095 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.136075    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c8d4n" podUID="d35415a6-2fc4-4544-8b27-cd22e3a87b00"
Sep 18 16:05:09.137884 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.137867    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 16:05:09.138398 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.138379    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vzt5f"
Sep 18 16:05:09.140404 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.140387    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 16:05:09.140510 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.140391    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6qnc4\""
Sep 18 16:05:09.140597 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.140516    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-26zfw"
Sep 18 16:05:09.140972 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.140955    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 16:05:09.142676 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.142515    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 16:05:09.142676 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.142609    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 16:05:09.142676 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.142638    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tjzzn\""
Sep 18 16:05:09.142896 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.142879    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 16:05:09.145722 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.145701    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9286k"
Sep 18 16:05:09.149038 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.148059    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 16:05:09.149038 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.148552    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.149038 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.148598    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kcclj\""
Sep 18 16:05:09.149038 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.148598    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 16:05:09.151298 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.150736    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret\") pod \"global-pull-secret-syncer-c8d4n\" (UID: \"d35415a6-2fc4-4544-8b27-cd22e3a87b00\") " pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:09.151298 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.150780    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf58bf07-5e75-4a2c-9168-29781769dec1-agent-certs\") pod \"konnectivity-agent-vzt5f\" (UID: \"cf58bf07-5e75-4a2c-9168-29781769dec1\") " pod="kube-system/konnectivity-agent-vzt5f"
Sep 18 16:05:09.151298 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.150785    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 16:05:09.151298 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.150813    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf58bf07-5e75-4a2c-9168-29781769dec1-konnectivity-ca\") pod \"konnectivity-agent-vzt5f\" (UID: \"cf58bf07-5e75-4a2c-9168-29781769dec1\") " pod="kube-system/konnectivity-agent-vzt5f"
Sep 18 16:05:09.151298 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.150851    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c78da59-56bd-4f33-a7e5-21f3f39e0446-system-cni-dir\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.151298 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.150889    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c78da59-56bd-4f33-a7e5-21f3f39e0446-cni-binary-copy\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.151298 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.150914    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhnlw\" (UniqueName: \"kubernetes.io/projected/1c5ec270-9dc8-45c4-bfb5-90f150a2fe54-kube-api-access-xhnlw\") pod \"iptables-alerter-f77cv\" (UID: \"1c5ec270-9dc8-45c4-bfb5-90f150a2fe54\") " pod="openshift-network-operator/iptables-alerter-f77cv"
Sep 18 16:05:09.151298 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.150962    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-log-socket\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.151298 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151032    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.151298 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151119    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7675747-b9b8-4eac-b41f-5eb3bf62d074-env-overrides\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.151298 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151185    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-sys-fs\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.151298 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151297    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-run-ovn\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.151949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151347    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-cni-netd\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.151949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151371    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7675747-b9b8-4eac-b41f-5eb3bf62d074-ovn-node-metrics-cert\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.151949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151397    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7675747-b9b8-4eac-b41f-5eb3bf62d074-ovnkube-script-lib\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.151949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151429    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1c78da59-56bd-4f33-a7e5-21f3f39e0446-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.151949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151455    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-slash\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.151949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151485    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-run-ovn-kubernetes\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.151949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151571    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c78da59-56bd-4f33-a7e5-21f3f39e0446-cnibin\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.151949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151609    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-device-dir\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.151949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151669    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5ec270-9dc8-45c4-bfb5-90f150a2fe54-host-slash\") pod \"iptables-alerter-f77cv\" (UID: \"1c5ec270-9dc8-45c4-bfb5-90f150a2fe54\") " pod="openshift-network-operator/iptables-alerter-f77cv"
Sep 18 16:05:09.151949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151722    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-kubelet\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.151949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151781    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-systemd-units\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.151949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151837    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-run-netns\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.151949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151871    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d35415a6-2fc4-4544-8b27-cd22e3a87b00-dbus\") pod \"global-pull-secret-syncer-c8d4n\" (UID: \"d35415a6-2fc4-4544-8b27-cd22e3a87b00\") " pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:09.151949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151911    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c78da59-56bd-4f33-a7e5-21f3f39e0446-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.151949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.151955    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1c78da59-56bd-4f33-a7e5-21f3f39e0446-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.152677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152023    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.152677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152065    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c5ec270-9dc8-45c4-bfb5-90f150a2fe54-iptables-alerter-script\") pod \"iptables-alerter-f77cv\" (UID: \"1c5ec270-9dc8-45c4-bfb5-90f150a2fe54\") " pod="openshift-network-operator/iptables-alerter-f77cv"
Sep 18 16:05:09.152677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152111    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-etc-openvswitch\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.152677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152150    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-run-openvswitch\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.152677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152179    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7675747-b9b8-4eac-b41f-5eb3bf62d074-ovnkube-config\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.152677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152202    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbs79\" (UniqueName: \"kubernetes.io/projected/1c78da59-56bd-4f33-a7e5-21f3f39e0446-kube-api-access-wbs79\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.152677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152241    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-socket-dir\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.152677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152258    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-etc-selinux\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.152677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152289    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkqxl\" (UniqueName: \"kubernetes.io/projected/c6df82d1-c147-481e-87bc-18da5165d85f-kube-api-access-tkqxl\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.152677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152349    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-run-systemd\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.152677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152398    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-var-lib-openvswitch\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.152677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152500    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-node-log\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.152677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152540    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:05:09.152677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152552    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8w8k\" (UniqueName: \"kubernetes.io/projected/e7675747-b9b8-4eac-b41f-5eb3bf62d074-kube-api-access-x8w8k\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.152677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152582    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-registration-dir\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.152677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152610    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-cni-bin\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.153343 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.152635    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d35415a6-2fc4-4544-8b27-cd22e3a87b00-kubelet-config\") pod \"global-pull-secret-syncer-c8d4n\" (UID: \"d35415a6-2fc4-4544-8b27-cd22e3a87b00\") " pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:09.154024 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.153995    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mdh8x\""
Sep 18 16:05:09.154329 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.154314    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.156321 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.156300    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:05:09.156604 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.156585    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 16:05:09.156781 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.156766    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bm6xr\""
Sep 18 16:05:09.157202 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.157176    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:09.157280 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.157249    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6psxt" podUID="dac4f70d-7789-4a02-8c3a-e2f28245e0ec"
Sep 18 16:05:09.159019 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.158998    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:09.159109 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.159059    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5sx5" podUID="e94bcc01-a5f8-49ef-ac80-07170e29c00d"
Sep 18 16:05:09.171758 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.171734    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p8b5q"
Sep 18 16:05:09.177961 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.177939    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p8b5q"
Sep 18 16:05:09.239317 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.239246    2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.252780    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c78da59-56bd-4f33-a7e5-21f3f39e0446-system-cni-dir\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.252819    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c78da59-56bd-4f33-a7e5-21f3f39e0446-cni-binary-copy\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.252841    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.252863    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8906b592-b7c8-46a7-ab14-c89b0803fd8d-hosts-file\") pod \"node-resolver-9286k\" (UID: \"8906b592-b7c8-46a7-ab14-c89b0803fd8d\") " pod="openshift-dns/node-resolver-9286k"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.252886    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-kubernetes\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.252901    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-lib-modules\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.252916    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-sys-fs\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.252937    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-run-ovn\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.252962    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7675747-b9b8-4eac-b41f-5eb3bf62d074-ovn-node-metrics-cert\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.252993    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7675747-b9b8-4eac-b41f-5eb3bf62d074-ovnkube-script-lib\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.253010    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-run-netns\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.253024    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-run\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.253043    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-sys\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.253061    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1c78da59-56bd-4f33-a7e5-21f3f39e0446-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.253084    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-slash\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.253104    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-sysctl-d\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.254553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.253136    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-multus-socket-dir-parent\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.253169    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-systemd-units\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.253188    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-run-netns\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.253616    2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.253749    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7675747-b9b8-4eac-b41f-5eb3bf62d074-ovnkube-script-lib\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254201    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1c78da59-56bd-4f33-a7e5-21f3f39e0446-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254253    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-slash\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254254    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c78da59-56bd-4f33-a7e5-21f3f39e0446-system-cni-dir\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254328    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-systemd-units\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254380    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254439    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-run-ovn\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254490    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-sys-fs\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254515    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d35415a6-2fc4-4544-8b27-cd22e3a87b00-dbus\") pod \"global-pull-secret-syncer-c8d4n\" (UID: \"d35415a6-2fc4-4544-8b27-cd22e3a87b00\") " pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254606    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g59wj\" (UniqueName: \"kubernetes.io/projected/8906b592-b7c8-46a7-ab14-c89b0803fd8d-kube-api-access-g59wj\") pod \"node-resolver-9286k\" (UID: \"8906b592-b7c8-46a7-ab14-c89b0803fd8d\") " pod="openshift-dns/node-resolver-9286k"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254631    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60f8bc85-96e6-4009-808d-914efbfd97e1-multus-daemon-config\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254652    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-var-lib-kubelet\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254680    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c78da59-56bd-4f33-a7e5-21f3f39e0446-cni-binary-copy\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.255429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254679    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c78da59-56bd-4f33-a7e5-21f3f39e0446-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254718    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254735    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-run-netns\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254742    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c5ec270-9dc8-45c4-bfb5-90f150a2fe54-iptables-alerter-script\") pod \"iptables-alerter-f77cv\" (UID: \"1c5ec270-9dc8-45c4-bfb5-90f150a2fe54\") " pod="openshift-network-operator/iptables-alerter-f77cv"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254788    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-etc-openvswitch\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254813    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-hostroot\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254844    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94qzl\" (UniqueName: \"kubernetes.io/projected/60f8bc85-96e6-4009-808d-914efbfd97e1-kube-api-access-94qzl\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254885    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-socket-dir\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254902    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d35415a6-2fc4-4544-8b27-cd22e3a87b00-dbus\") pod \"global-pull-secret-syncer-c8d4n\" (UID: \"d35415a6-2fc4-4544-8b27-cd22e3a87b00\") " pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254912    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-etc-selinux\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254902    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.254937    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8w8k\" (UniqueName: \"kubernetes.io/projected/e7675747-b9b8-4eac-b41f-5eb3bf62d074-kube-api-access-x8w8k\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255051    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c78da59-56bd-4f33-a7e5-21f3f39e0446-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255098    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-socket-dir\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255190    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf58bf07-5e75-4a2c-9168-29781769dec1-agent-certs\") pod \"konnectivity-agent-vzt5f\" (UID: \"cf58bf07-5e75-4a2c-9168-29781769dec1\") " pod="kube-system/konnectivity-agent-vzt5f"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255228    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8906b592-b7c8-46a7-ab14-c89b0803fd8d-tmp-dir\") pod \"node-resolver-9286k\" (UID: \"8906b592-b7c8-46a7-ab14-c89b0803fd8d\") " pod="openshift-dns/node-resolver-9286k"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255271    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60f8bc85-96e6-4009-808d-914efbfd97e1-cni-binary-copy\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.256281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255333    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-run-k8s-cni-cncf-io\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255375    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-sysctl-conf\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255430    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-registration-dir\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255457    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-cni-bin\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255463    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-etc-selinux\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255483    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d35415a6-2fc4-4544-8b27-cd22e3a87b00-kubelet-config\") pod \"global-pull-secret-syncer-c8d4n\" (UID: \"d35415a6-2fc4-4544-8b27-cd22e3a87b00\") " pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255556    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf58bf07-5e75-4a2c-9168-29781769dec1-konnectivity-ca\") pod \"konnectivity-agent-vzt5f\" (UID: \"cf58bf07-5e75-4a2c-9168-29781769dec1\") " pod="kube-system/konnectivity-agent-vzt5f"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255586    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-var-lib-cni-multus\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255611    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-systemd\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255643    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhnlw\" (UniqueName: \"kubernetes.io/projected/1c5ec270-9dc8-45c4-bfb5-90f150a2fe54-kube-api-access-xhnlw\") pod \"iptables-alerter-f77cv\" (UID: \"1c5ec270-9dc8-45c4-bfb5-90f150a2fe54\") " pod="openshift-network-operator/iptables-alerter-f77cv"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255643    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c5ec270-9dc8-45c4-bfb5-90f150a2fe54-iptables-alerter-script\") pod \"iptables-alerter-f77cv\" (UID: \"1c5ec270-9dc8-45c4-bfb5-90f150a2fe54\") " pod="openshift-network-operator/iptables-alerter-f77cv"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255671    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-log-socket\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255704    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7675747-b9b8-4eac-b41f-5eb3bf62d074-env-overrides\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255724    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d35415a6-2fc4-4544-8b27-cd22e3a87b00-kubelet-config\") pod \"global-pull-secret-syncer-c8d4n\" (UID: \"d35415a6-2fc4-4544-8b27-cd22e3a87b00\") " pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255725    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-system-cni-dir\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255768    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-etc-openvswitch\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255783    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-registration-dir\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.257107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.255849    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-cni-bin\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.256119    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-log-socket\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.256451    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf58bf07-5e75-4a2c-9168-29781769dec1-konnectivity-ca\") pod \"konnectivity-agent-vzt5f\" (UID: \"cf58bf07-5e75-4a2c-9168-29781769dec1\") " pod="kube-system/konnectivity-agent-vzt5f"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.256503    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-var-lib-kubelet\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.256590    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-run-multus-certs\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.256674    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-modprobe-d\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.256712    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-tuned\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.256755    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-cni-netd\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.256778    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4d012abb-e750-4643-b3fe-8cffd9723e76-serviceca\") pod \"node-ca-26zfw\" (UID: \"4d012abb-e750-4643-b3fe-8cffd9723e76\") " pod="openshift-image-registry/node-ca-26zfw"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.256814    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dsv2\" (UniqueName: \"kubernetes.io/projected/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-kube-api-access-7dsv2\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.256849    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-run-ovn-kubernetes\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.256885    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d012abb-e750-4643-b3fe-8cffd9723e76-host\") pod \"node-ca-26zfw\" (UID: \"4d012abb-e750-4643-b3fe-8cffd9723e76\") " pod="openshift-image-registry/node-ca-26zfw"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.256915    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk59c\" (UniqueName: \"kubernetes.io/projected/4d012abb-e750-4643-b3fe-8cffd9723e76-kube-api-access-lk59c\") pod \"node-ca-26zfw\" (UID: \"4d012abb-e750-4643-b3fe-8cffd9723e76\") " pod="openshift-image-registry/node-ca-26zfw"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.256938    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-sysconfig\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.256972    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-tmp\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.256987    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7675747-b9b8-4eac-b41f-5eb3bf62d074-env-overrides\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.256998    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c78da59-56bd-4f33-a7e5-21f3f39e0446-cnibin\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.257989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257032    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-device-dir\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257055    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5ec270-9dc8-45c4-bfb5-90f150a2fe54-host-slash\") pod \"iptables-alerter-f77cv\" (UID: \"1c5ec270-9dc8-45c4-bfb5-90f150a2fe54\") " pod="openshift-network-operator/iptables-alerter-f77cv"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257071    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-cni-netd\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257111    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-kubelet\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257135    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6df82d1-c147-481e-87bc-18da5165d85f-device-dir\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257189    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-kubelet\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257184    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-var-lib-cni-bin\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257235    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-multus-conf-dir\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257268    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-host-run-ovn-kubernetes\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257238    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c78da59-56bd-4f33-a7e5-21f3f39e0446-cnibin\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257274    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnd69\" (UniqueName: \"kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69\") pod \"network-check-target-j5sx5\" (UID: \"e94bcc01-a5f8-49ef-ac80-07170e29c00d\") " pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257292    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7675747-b9b8-4eac-b41f-5eb3bf62d074-ovn-node-metrics-cert\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257331    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5ec270-9dc8-45c4-bfb5-90f150a2fe54-host-slash\") pod \"iptables-alerter-f77cv\" (UID: \"1c5ec270-9dc8-45c4-bfb5-90f150a2fe54\") " pod="openshift-network-operator/iptables-alerter-f77cv"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257356    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1c78da59-56bd-4f33-a7e5-21f3f39e0446-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257388    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-run-openvswitch\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257413    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-cnibin\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257438    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7675747-b9b8-4eac-b41f-5eb3bf62d074-ovnkube-config\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.258677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257452    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-run-openvswitch\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257493    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-multus-cni-dir\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257542    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-etc-kubernetes\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257750    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbs79\" (UniqueName: \"kubernetes.io/projected/1c78da59-56bd-4f33-a7e5-21f3f39e0446-kube-api-access-wbs79\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257786    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkqxl\" (UniqueName: \"kubernetes.io/projected/c6df82d1-c147-481e-87bc-18da5165d85f-kube-api-access-tkqxl\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257810    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-run-systemd\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257848    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-var-lib-openvswitch\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257871    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-node-log\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257895    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret\") pod \"global-pull-secret-syncer-c8d4n\" (UID: \"d35415a6-2fc4-4544-8b27-cd22e3a87b00\") " pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257935    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-host\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257958    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs\") pod \"network-metrics-daemon-6psxt\" (UID: \"dac4f70d-7789-4a02-8c3a-e2f28245e0ec\") " pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.257981    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxznc\" (UniqueName: \"kubernetes.io/projected/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-kube-api-access-dxznc\") pod \"network-metrics-daemon-6psxt\" (UID: \"dac4f70d-7789-4a02-8c3a-e2f28245e0ec\") " pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.258010    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7675747-b9b8-4eac-b41f-5eb3bf62d074-ovnkube-config\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.258070    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-var-lib-openvswitch\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.258073    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-node-log\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.258133    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1c78da59-56bd-4f33-a7e5-21f3f39e0446-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.258168    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:09.259421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.258209    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7675747-b9b8-4eac-b41f-5eb3bf62d074-run-systemd\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.260063 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.258251    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret podName:d35415a6-2fc4-4544-8b27-cd22e3a87b00 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:09.758210175 +0000 UTC m=+4.705708953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret") pod "global-pull-secret-syncer-c8d4n" (UID: "d35415a6-2fc4-4544-8b27-cd22e3a87b00") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:09.260063 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.258700    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf58bf07-5e75-4a2c-9168-29781769dec1-agent-certs\") pod \"konnectivity-agent-vzt5f\" (UID: \"cf58bf07-5e75-4a2c-9168-29781769dec1\") " pod="kube-system/konnectivity-agent-vzt5f"
Sep 18 16:05:09.263835 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.263811    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8w8k\" (UniqueName: \"kubernetes.io/projected/e7675747-b9b8-4eac-b41f-5eb3bf62d074-kube-api-access-x8w8k\") pod \"ovnkube-node-f79kb\" (UID: \"e7675747-b9b8-4eac-b41f-5eb3bf62d074\") " pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.264575 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.264550    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhnlw\" (UniqueName: \"kubernetes.io/projected/1c5ec270-9dc8-45c4-bfb5-90f150a2fe54-kube-api-access-xhnlw\") pod \"iptables-alerter-f77cv\" (UID: \"1c5ec270-9dc8-45c4-bfb5-90f150a2fe54\") " pod="openshift-network-operator/iptables-alerter-f77cv"
Sep 18 16:05:09.266459 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.266442    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbs79\" (UniqueName: \"kubernetes.io/projected/1c78da59-56bd-4f33-a7e5-21f3f39e0446-kube-api-access-wbs79\") pod \"multus-additional-cni-plugins-nw2sh\" (UID: \"1c78da59-56bd-4f33-a7e5-21f3f39e0446\") " pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.274918 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.274873    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkqxl\" (UniqueName: \"kubernetes.io/projected/c6df82d1-c147-481e-87bc-18da5165d85f-kube-api-access-tkqxl\") pod \"aws-ebs-csi-driver-node-65fqj\" (UID: \"c6df82d1-c147-481e-87bc-18da5165d85f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.277533 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.277474    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal" event={"ID":"1ea19d78c8a13e00dcd379551f4b0152","Type":"ContainerStarted","Data":"25999dfbe78caa8fb58751ddfba7e0c86f6ce7660e2054fa2e59ec5ef981aebb"}
Sep 18 16:05:09.279592 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.279561    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.ec2.internal" event={"ID":"f8694b5494e0c8ea7c82d6e3efeab8b4","Type":"ContainerStarted","Data":"5c8343d081ba925800ad8e3b97be7365e546fa9ced41ea7eb7f476a188a716a6"}
Sep 18 16:05:09.358743 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.358654    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs\") pod \"network-metrics-daemon-6psxt\" (UID: \"dac4f70d-7789-4a02-8c3a-e2f28245e0ec\") " pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:09.358743 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.358706    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxznc\" (UniqueName: \"kubernetes.io/projected/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-kube-api-access-dxznc\") pod \"network-metrics-daemon-6psxt\" (UID: \"dac4f70d-7789-4a02-8c3a-e2f28245e0ec\") " pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:09.358743 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.358728    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8906b592-b7c8-46a7-ab14-c89b0803fd8d-hosts-file\") pod \"node-resolver-9286k\" (UID: \"8906b592-b7c8-46a7-ab14-c89b0803fd8d\") " pod="openshift-dns/node-resolver-9286k"
Sep 18 16:05:09.359015 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.358749    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-kubernetes\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.359015 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.358767    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:09.359015 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.358815    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-kubernetes\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.359015 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.358827    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs podName:dac4f70d-7789-4a02-8c3a-e2f28245e0ec nodeName:}" failed. No retries permitted until 2026-09-18 16:05:09.858811324 +0000 UTC m=+4.806310080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs") pod "network-metrics-daemon-6psxt" (UID: "dac4f70d-7789-4a02-8c3a-e2f28245e0ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:09.359015 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.358839    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8906b592-b7c8-46a7-ab14-c89b0803fd8d-hosts-file\") pod \"node-resolver-9286k\" (UID: \"8906b592-b7c8-46a7-ab14-c89b0803fd8d\") " pod="openshift-dns/node-resolver-9286k"
Sep 18 16:05:09.359015 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.358878    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-lib-modules\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.359015 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.358917    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-run-netns\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.359015 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.358940    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-run\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.359015 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.358959    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-sys\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.359015 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.358969    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-run-netns\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.359015 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.358979    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-sysctl-d\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.359015 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359003    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-multus-socket-dir-parent\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.359015 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359021    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-run\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359027    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g59wj\" (UniqueName: \"kubernetes.io/projected/8906b592-b7c8-46a7-ab14-c89b0803fd8d-kube-api-access-g59wj\") pod \"node-resolver-9286k\" (UID: \"8906b592-b7c8-46a7-ab14-c89b0803fd8d\") " pod="openshift-dns/node-resolver-9286k"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359057    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-lib-modules\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359062    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-sys\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359079    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-multus-socket-dir-parent\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359113    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60f8bc85-96e6-4009-808d-914efbfd97e1-multus-daemon-config\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359125    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-sysctl-d\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359139    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-var-lib-kubelet\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359171    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-hostroot\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359187    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-var-lib-kubelet\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359194    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qzl\" (UniqueName: \"kubernetes.io/projected/60f8bc85-96e6-4009-808d-914efbfd97e1-kube-api-access-94qzl\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359220    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8906b592-b7c8-46a7-ab14-c89b0803fd8d-tmp-dir\") pod \"node-resolver-9286k\" (UID: \"8906b592-b7c8-46a7-ab14-c89b0803fd8d\") " pod="openshift-dns/node-resolver-9286k"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359228    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-hostroot\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359241    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60f8bc85-96e6-4009-808d-914efbfd97e1-cni-binary-copy\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359385    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-run-k8s-cni-cncf-io\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359405    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-sysctl-conf\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359421    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-var-lib-cni-multus\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359435    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-systemd\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.359716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359450    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-system-cni-dir\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359463    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-var-lib-kubelet\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359478    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-run-multus-certs\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359493    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-modprobe-d\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359498    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8906b592-b7c8-46a7-ab14-c89b0803fd8d-tmp-dir\") pod \"node-resolver-9286k\" (UID: \"8906b592-b7c8-46a7-ab14-c89b0803fd8d\") " pod="openshift-dns/node-resolver-9286k"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359508    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-tuned\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359551    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4d012abb-e750-4643-b3fe-8cffd9723e76-serviceca\") pod \"node-ca-26zfw\" (UID: \"4d012abb-e750-4643-b3fe-8cffd9723e76\") " pod="openshift-image-registry/node-ca-26zfw"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359576    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dsv2\" (UniqueName: \"kubernetes.io/projected/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-kube-api-access-7dsv2\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359602    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d012abb-e750-4643-b3fe-8cffd9723e76-host\") pod \"node-ca-26zfw\" (UID: \"4d012abb-e750-4643-b3fe-8cffd9723e76\") " pod="openshift-image-registry/node-ca-26zfw"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359620    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-system-cni-dir\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359625    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk59c\" (UniqueName: \"kubernetes.io/projected/4d012abb-e750-4643-b3fe-8cffd9723e76-kube-api-access-lk59c\") pod \"node-ca-26zfw\" (UID: \"4d012abb-e750-4643-b3fe-8cffd9723e76\") " pod="openshift-image-registry/node-ca-26zfw"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359671    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-sysconfig\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359696    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-tmp\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359724    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-var-lib-cni-bin\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359736    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60f8bc85-96e6-4009-808d-914efbfd97e1-multus-daemon-config\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359750    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-multus-conf-dir\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359774    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnd69\" (UniqueName: \"kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69\") pod \"network-check-target-j5sx5\" (UID: \"e94bcc01-a5f8-49ef-ac80-07170e29c00d\") " pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359777    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60f8bc85-96e6-4009-808d-914efbfd97e1-cni-binary-copy\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.360546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359838    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-cnibin\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359854    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-systemd\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359867    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-multus-cni-dir\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359892    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-etc-kubernetes\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359898    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-sysconfig\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359958    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-var-lib-kubelet\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359995    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-run-multus-certs\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.359998    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d012abb-e750-4643-b3fe-8cffd9723e76-host\") pod \"node-ca-26zfw\" (UID: \"4d012abb-e750-4643-b3fe-8cffd9723e76\") " pod="openshift-image-registry/node-ca-26zfw"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.360041    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-sysctl-conf\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.360089    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-run-k8s-cni-cncf-io\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.360103    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-modprobe-d\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.360128    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-multus-conf-dir\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.360157    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-host\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.360166    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-var-lib-cni-bin\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.360240    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-host-var-lib-cni-multus\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.360242    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-host\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.360293    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-cnibin\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.360288    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-multus-cni-dir\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.361318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.360311    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60f8bc85-96e6-4009-808d-914efbfd97e1-etc-kubernetes\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.362106 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.360358    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4d012abb-e750-4643-b3fe-8cffd9723e76-serviceca\") pod \"node-ca-26zfw\" (UID: \"4d012abb-e750-4643-b3fe-8cffd9723e76\") " pod="openshift-image-registry/node-ca-26zfw"
Sep 18 16:05:09.362849 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.362820    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-tmp\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.364252 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.364229    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-etc-tuned\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.365831 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.365808    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:05:09.365831 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.365839    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:05:09.366088 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.365853    2554 projected.go:196] Error preparing data for projected volume kube-api-access-xnd69 for pod openshift-network-diagnostics/network-check-target-j5sx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:09.366088 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.365923    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69 podName:e94bcc01-a5f8-49ef-ac80-07170e29c00d nodeName:}" failed. No retries permitted until 2026-09-18 16:05:09.86590452 +0000 UTC m=+4.813403288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xnd69" (UniqueName: "kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69") pod "network-check-target-j5sx5" (UID: "e94bcc01-a5f8-49ef-ac80-07170e29c00d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:09.368287 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.368263    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94qzl\" (UniqueName: \"kubernetes.io/projected/60f8bc85-96e6-4009-808d-914efbfd97e1-kube-api-access-94qzl\") pod \"multus-w26s7\" (UID: \"60f8bc85-96e6-4009-808d-914efbfd97e1\") " pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.368690 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.368666    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dsv2\" (UniqueName: \"kubernetes.io/projected/ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed-kube-api-access-7dsv2\") pod \"tuned-wwwrj\" (UID: \"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed\") " pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.368786 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.368708    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxznc\" (UniqueName: \"kubernetes.io/projected/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-kube-api-access-dxznc\") pod \"network-metrics-daemon-6psxt\" (UID: \"dac4f70d-7789-4a02-8c3a-e2f28245e0ec\") " pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:09.369684 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.369656    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g59wj\" (UniqueName: \"kubernetes.io/projected/8906b592-b7c8-46a7-ab14-c89b0803fd8d-kube-api-access-g59wj\") pod \"node-resolver-9286k\" (UID: \"8906b592-b7c8-46a7-ab14-c89b0803fd8d\") " pod="openshift-dns/node-resolver-9286k"
Sep 18 16:05:09.370162 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.370142    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk59c\" (UniqueName: \"kubernetes.io/projected/4d012abb-e750-4643-b3fe-8cffd9723e76-kube-api-access-lk59c\") pod \"node-ca-26zfw\" (UID: \"4d012abb-e750-4643-b3fe-8cffd9723e76\") " pod="openshift-image-registry/node-ca-26zfw"
Sep 18 16:05:09.440124 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.440085    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nw2sh"
Sep 18 16:05:09.454023 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.453983    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f77cv"
Sep 18 16:05:09.464119 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.464096    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj"
Sep 18 16:05:09.470765 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.470745    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:09.477328 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.477310    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vzt5f"
Sep 18 16:05:09.484905 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.484884    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-26zfw"
Sep 18 16:05:09.492405 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.492382    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9286k"
Sep 18 16:05:09.499077 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.499057    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w26s7"
Sep 18 16:05:09.504740 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.504721    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wwwrj"
Sep 18 16:05:09.559497 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.559465    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:05:09.763459 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.763372    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret\") pod \"global-pull-secret-syncer-c8d4n\" (UID: \"d35415a6-2fc4-4544-8b27-cd22e3a87b00\") " pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:09.763622 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.763565    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:09.763691 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.763648    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret podName:d35415a6-2fc4-4544-8b27-cd22e3a87b00 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:10.763627298 +0000 UTC m=+5.711126063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret") pod "global-pull-secret-syncer-c8d4n" (UID: "d35415a6-2fc4-4544-8b27-cd22e3a87b00") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:09.864029 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.863989    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs\") pod \"network-metrics-daemon-6psxt\" (UID: \"dac4f70d-7789-4a02-8c3a-e2f28245e0ec\") " pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:09.864210 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.864122    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:09.864285 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.864217    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs podName:dac4f70d-7789-4a02-8c3a-e2f28245e0ec nodeName:}" failed. No retries permitted until 2026-09-18 16:05:10.864191239 +0000 UTC m=+5.811690007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs") pod "network-metrics-daemon-6psxt" (UID: "dac4f70d-7789-4a02-8c3a-e2f28245e0ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:09.964540 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:09.964489    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnd69\" (UniqueName: \"kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69\") pod \"network-check-target-j5sx5\" (UID: \"e94bcc01-a5f8-49ef-ac80-07170e29c00d\") " pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:09.964676 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.964641    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:05:09.964676 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.964660    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:05:09.964676 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.964669    2554 projected.go:196] Error preparing data for projected volume kube-api-access-xnd69 for pod openshift-network-diagnostics/network-check-target-j5sx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:09.964802 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:09.964731    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69 podName:e94bcc01-a5f8-49ef-ac80-07170e29c00d nodeName:}" failed. No retries permitted until 2026-09-18 16:05:10.964709031 +0000 UTC m=+5.912207784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xnd69" (UniqueName: "kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69") pod "network-check-target-j5sx5" (UID: "e94bcc01-a5f8-49ef-ac80-07170e29c00d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:10.180840 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:10.180757    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:00:09 +0000 UTC" deadline="2028-04-01 01:32:57.543192332 +0000 UTC"
Sep 18 16:05:10.180840 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:10.180794    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13449h27m47.362402543s"
Sep 18 16:05:10.307146 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:10.307113    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5ec270_9dc8_45c4_bfb5_90f150a2fe54.slice/crio-27d7808562f9e003d96260e34eac75188d1b006a0879da5b2ad39d99cfeceb63 WatchSource:0}: Error finding container 27d7808562f9e003d96260e34eac75188d1b006a0879da5b2ad39d99cfeceb63: Status 404 returned error can't find the container with id 27d7808562f9e003d96260e34eac75188d1b006a0879da5b2ad39d99cfeceb63
Sep 18 16:05:10.307958 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:10.307930    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf58bf07_5e75_4a2c_9168_29781769dec1.slice/crio-3d9db9951b054917aec95e31952894f385ef19e5a207344ab9ae5c20fdfd2fd6 WatchSource:0}: Error finding container 3d9db9951b054917aec95e31952894f385ef19e5a207344ab9ae5c20fdfd2fd6: Status 404 returned error can't find the container with id 3d9db9951b054917aec95e31952894f385ef19e5a207344ab9ae5c20fdfd2fd6
Sep 18 16:05:10.310680 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:10.310654    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c78da59_56bd_4f33_a7e5_21f3f39e0446.slice/crio-73bd4ca11e9bb717d10203bfcace2c557e9dafd8be2dd461b264f7916bba1c8d WatchSource:0}: Error finding container 73bd4ca11e9bb717d10203bfcace2c557e9dafd8be2dd461b264f7916bba1c8d: Status 404 returned error can't find the container with id 73bd4ca11e9bb717d10203bfcace2c557e9dafd8be2dd461b264f7916bba1c8d
Sep 18 16:05:10.313376 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:10.313181    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f8bc85_96e6_4009_808d_914efbfd97e1.slice/crio-8b928d9cb0082705a01b2499688a7af38ee171bc6b844dff4cb5f9d19a3acd3b WatchSource:0}: Error finding container 8b928d9cb0082705a01b2499688a7af38ee171bc6b844dff4cb5f9d19a3acd3b: Status 404 returned error can't find the container with id 8b928d9cb0082705a01b2499688a7af38ee171bc6b844dff4cb5f9d19a3acd3b
Sep 18 16:05:10.315205 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:10.315186    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8906b592_b7c8_46a7_ab14_c89b0803fd8d.slice/crio-77b154dd157ce7a631b848213a07affb242d2fb0ec530c415e41f359c40cfbfe WatchSource:0}: Error finding container 77b154dd157ce7a631b848213a07affb242d2fb0ec530c415e41f359c40cfbfe: Status 404 returned error can't find the container with id 77b154dd157ce7a631b848213a07affb242d2fb0ec530c415e41f359c40cfbfe
Sep 18 16:05:10.316088 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:10.316052    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6df82d1_c147_481e_87bc_18da5165d85f.slice/crio-fd3f495b19d9c1abe3bfede4531eab101475b9dfd694475f8e81bb7ed15f3c2d WatchSource:0}: Error finding container fd3f495b19d9c1abe3bfede4531eab101475b9dfd694475f8e81bb7ed15f3c2d: Status 404 returned error can't find the container with id fd3f495b19d9c1abe3bfede4531eab101475b9dfd694475f8e81bb7ed15f3c2d
Sep 18 16:05:10.317571 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:10.317505    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7675747_b9b8_4eac_b41f_5eb3bf62d074.slice/crio-0fc80de1e8116171f36417e647187e90a017e4979f7bd456667fe73a57fd68ed WatchSource:0}: Error finding container 0fc80de1e8116171f36417e647187e90a017e4979f7bd456667fe73a57fd68ed: Status 404 returned error can't find the container with id 0fc80de1e8116171f36417e647187e90a017e4979f7bd456667fe73a57fd68ed
Sep 18 16:05:10.318826 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:10.318398    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d012abb_e750_4643_b3fe_8cffd9723e76.slice/crio-47f066581e1703cb622e5105ed867031dacb961dc26bb8ecb647a5220d72c095 WatchSource:0}: Error finding container 47f066581e1703cb622e5105ed867031dacb961dc26bb8ecb647a5220d72c095: Status 404 returned error can't find the container with id 47f066581e1703cb622e5105ed867031dacb961dc26bb8ecb647a5220d72c095
Sep 18 16:05:10.319637 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:10.319608    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7e13d9_6ffe_47ec_8e85_a19bd587f9ed.slice/crio-c69cffdd3c5045ce2a36f00acd5c09840311f555aa3bea25eab61ddfbe00e49e WatchSource:0}: Error finding container c69cffdd3c5045ce2a36f00acd5c09840311f555aa3bea25eab61ddfbe00e49e: Status 404 returned error can't find the container with id c69cffdd3c5045ce2a36f00acd5c09840311f555aa3bea25eab61ddfbe00e49e
Sep 18 16:05:10.770397 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:10.769998    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret\") pod \"global-pull-secret-syncer-c8d4n\" (UID: \"d35415a6-2fc4-4544-8b27-cd22e3a87b00\") " pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:10.770397 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:10.770265    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:10.770397 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:10.770341    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret podName:d35415a6-2fc4-4544-8b27-cd22e3a87b00 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:12.770320686 +0000 UTC m=+7.717819438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret") pod "global-pull-secret-syncer-c8d4n" (UID: "d35415a6-2fc4-4544-8b27-cd22e3a87b00") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:10.870991 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:10.870934    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs\") pod \"network-metrics-daemon-6psxt\" (UID: \"dac4f70d-7789-4a02-8c3a-e2f28245e0ec\") " pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:10.871175 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:10.871125    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:10.871239 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:10.871190    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs podName:dac4f70d-7789-4a02-8c3a-e2f28245e0ec nodeName:}" failed. No retries permitted until 2026-09-18 16:05:12.871171769 +0000 UTC m=+7.818670521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs") pod "network-metrics-daemon-6psxt" (UID: "dac4f70d-7789-4a02-8c3a-e2f28245e0ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:10.972877 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:10.972191    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnd69\" (UniqueName: \"kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69\") pod \"network-check-target-j5sx5\" (UID: \"e94bcc01-a5f8-49ef-ac80-07170e29c00d\") " pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:10.972877 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:10.972403    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:05:10.972877 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:10.972423    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:05:10.972877 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:10.972435    2554 projected.go:196] Error preparing data for projected volume kube-api-access-xnd69 for pod openshift-network-diagnostics/network-check-target-j5sx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:10.972877 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:10.972494    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69 podName:e94bcc01-a5f8-49ef-ac80-07170e29c00d nodeName:}" failed. No retries permitted until 2026-09-18 16:05:12.972474165 +0000 UTC m=+7.919972920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xnd69" (UniqueName: "kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69") pod "network-check-target-j5sx5" (UID: "e94bcc01-a5f8-49ef-ac80-07170e29c00d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:11.181658 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.181545    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:00:09 +0000 UTC" deadline="2028-06-05 16:36:48.350549588 +0000 UTC"
Sep 18 16:05:11.181658 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.181607    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15024h31m37.168946787s"
Sep 18 16:05:11.274591 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.273647    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:11.274591 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:11.273783    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6psxt" podUID="dac4f70d-7789-4a02-8c3a-e2f28245e0ec"
Sep 18 16:05:11.274591 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.274432    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:11.274591 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:11.274545    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5sx5" podUID="e94bcc01-a5f8-49ef-ac80-07170e29c00d"
Sep 18 16:05:11.275106 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.274982    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:11.275106 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:11.275074    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c8d4n" podUID="d35415a6-2fc4-4544-8b27-cd22e3a87b00"
Sep 18 16:05:11.304963 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.304929    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vzt5f" event={"ID":"cf58bf07-5e75-4a2c-9168-29781769dec1","Type":"ContainerStarted","Data":"3d9db9951b054917aec95e31952894f385ef19e5a207344ab9ae5c20fdfd2fd6"}
Sep 18 16:05:11.330584 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.330545    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f77cv" event={"ID":"1c5ec270-9dc8-45c4-bfb5-90f150a2fe54","Type":"ContainerStarted","Data":"27d7808562f9e003d96260e34eac75188d1b006a0879da5b2ad39d99cfeceb63"}
Sep 18 16:05:11.335001 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.334970    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.ec2.internal" event={"ID":"f8694b5494e0c8ea7c82d6e3efeab8b4","Type":"ContainerStarted","Data":"0eb7372006b2a4831ded4dfb8c618b6ec29da3b4f4cd5cbc61dd1b8afde89b0f"}
Sep 18 16:05:11.336875 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.336839    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb" event={"ID":"e7675747-b9b8-4eac-b41f-5eb3bf62d074","Type":"ContainerStarted","Data":"0fc80de1e8116171f36417e647187e90a017e4979f7bd456667fe73a57fd68ed"}
Sep 18 16:05:11.340150 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.340080    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nw2sh" event={"ID":"1c78da59-56bd-4f33-a7e5-21f3f39e0446","Type":"ContainerStarted","Data":"73bd4ca11e9bb717d10203bfcace2c557e9dafd8be2dd461b264f7916bba1c8d"}
Sep 18 16:05:11.353346 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.352386    2554 generic.go:356] "Generic (PLEG): container finished" podID="1ea19d78c8a13e00dcd379551f4b0152" containerID="3adf6a554eea6cbfdde16a94b4dfd052c3c57cfc5f15a300864ea3443eb059c8" exitCode=0
Sep 18 16:05:11.353346 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.352463    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal" event={"ID":"1ea19d78c8a13e00dcd379551f4b0152","Type":"ContainerDied","Data":"3adf6a554eea6cbfdde16a94b4dfd052c3c57cfc5f15a300864ea3443eb059c8"}
Sep 18 16:05:11.367988 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.367924    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wwwrj" event={"ID":"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed","Type":"ContainerStarted","Data":"c69cffdd3c5045ce2a36f00acd5c09840311f555aa3bea25eab61ddfbe00e49e"}
Sep 18 16:05:11.368436 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.368344    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-195.ec2.internal" podStartSLOduration=3.368325129 podStartE2EDuration="3.368325129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:11.351800015 +0000 UTC m=+6.299298790" watchObservedRunningTime="2026-09-18 16:05:11.368325129 +0000 UTC m=+6.315823904"
Sep 18 16:05:11.378511 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.378410    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-26zfw" event={"ID":"4d012abb-e750-4643-b3fe-8cffd9723e76","Type":"ContainerStarted","Data":"47f066581e1703cb622e5105ed867031dacb961dc26bb8ecb647a5220d72c095"}
Sep 18 16:05:11.390491 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.390457    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj" event={"ID":"c6df82d1-c147-481e-87bc-18da5165d85f","Type":"ContainerStarted","Data":"fd3f495b19d9c1abe3bfede4531eab101475b9dfd694475f8e81bb7ed15f3c2d"}
Sep 18 16:05:11.399348 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.399313    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9286k" event={"ID":"8906b592-b7c8-46a7-ab14-c89b0803fd8d","Type":"ContainerStarted","Data":"77b154dd157ce7a631b848213a07affb242d2fb0ec530c415e41f359c40cfbfe"}
Sep 18 16:05:11.402366 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:11.402333    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w26s7" event={"ID":"60f8bc85-96e6-4009-808d-914efbfd97e1","Type":"ContainerStarted","Data":"8b928d9cb0082705a01b2499688a7af38ee171bc6b844dff4cb5f9d19a3acd3b"}
Sep 18 16:05:12.420690 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:12.420622    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal" event={"ID":"1ea19d78c8a13e00dcd379551f4b0152","Type":"ContainerStarted","Data":"51893ab071e022c83199b4963f03d73b5bde6a430691fab332020022f9c2349b"}
Sep 18 16:05:12.438294 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:12.437467    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-195.ec2.internal" podStartSLOduration=4.43744821 podStartE2EDuration="4.43744821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:12.437047829 +0000 UTC m=+7.384546620" watchObservedRunningTime="2026-09-18 16:05:12.43744821 +0000 UTC m=+7.384946987"
Sep 18 16:05:12.789047 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:12.788312    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret\") pod \"global-pull-secret-syncer-c8d4n\" (UID: \"d35415a6-2fc4-4544-8b27-cd22e3a87b00\") " pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:12.789047 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:12.788562    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:12.789047 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:12.788634    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret podName:d35415a6-2fc4-4544-8b27-cd22e3a87b00 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:16.788613484 +0000 UTC m=+11.736112249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret") pod "global-pull-secret-syncer-c8d4n" (UID: "d35415a6-2fc4-4544-8b27-cd22e3a87b00") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:12.888894 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:12.888854    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs\") pod \"network-metrics-daemon-6psxt\" (UID: \"dac4f70d-7789-4a02-8c3a-e2f28245e0ec\") " pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:12.889092 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:12.889039    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:12.889149 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:12.889103    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs podName:dac4f70d-7789-4a02-8c3a-e2f28245e0ec nodeName:}" failed. No retries permitted until 2026-09-18 16:05:16.889085044 +0000 UTC m=+11.836583813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs") pod "network-metrics-daemon-6psxt" (UID: "dac4f70d-7789-4a02-8c3a-e2f28245e0ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:12.990757 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:12.990111    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnd69\" (UniqueName: \"kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69\") pod \"network-check-target-j5sx5\" (UID: \"e94bcc01-a5f8-49ef-ac80-07170e29c00d\") " pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:12.990757 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:12.990266    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:05:12.990757 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:12.990293    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:05:12.990757 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:12.990307    2554 projected.go:196] Error preparing data for projected volume kube-api-access-xnd69 for pod openshift-network-diagnostics/network-check-target-j5sx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:12.990757 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:12.990365    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69 podName:e94bcc01-a5f8-49ef-ac80-07170e29c00d nodeName:}" failed. No retries permitted until 2026-09-18 16:05:16.990346821 +0000 UTC m=+11.937845577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xnd69" (UniqueName: "kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69") pod "network-check-target-j5sx5" (UID: "e94bcc01-a5f8-49ef-ac80-07170e29c00d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:13.272147 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:13.271658    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:13.272147 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:13.271668    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:13.272147 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:13.271801    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6psxt" podUID="dac4f70d-7789-4a02-8c3a-e2f28245e0ec"
Sep 18 16:05:13.272147 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:13.271854    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5sx5" podUID="e94bcc01-a5f8-49ef-ac80-07170e29c00d"
Sep 18 16:05:13.272147 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:13.271937    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:13.272147 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:13.272006    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c8d4n" podUID="d35415a6-2fc4-4544-8b27-cd22e3a87b00"
Sep 18 16:05:15.272312 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:15.271641    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:15.272312 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:15.271778    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c8d4n" podUID="d35415a6-2fc4-4544-8b27-cd22e3a87b00"
Sep 18 16:05:15.272312 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:15.272153    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:15.272312 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:15.272255    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6psxt" podUID="dac4f70d-7789-4a02-8c3a-e2f28245e0ec"
Sep 18 16:05:15.273200 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:15.272594    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:15.273200 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:15.272699    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5sx5" podUID="e94bcc01-a5f8-49ef-ac80-07170e29c00d"
Sep 18 16:05:16.823590 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:16.823551    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret\") pod \"global-pull-secret-syncer-c8d4n\" (UID: \"d35415a6-2fc4-4544-8b27-cd22e3a87b00\") " pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:16.824054 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:16.823739    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:16.824054 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:16.823841    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret podName:d35415a6-2fc4-4544-8b27-cd22e3a87b00 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:24.823822249 +0000 UTC m=+19.771321002 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret") pod "global-pull-secret-syncer-c8d4n" (UID: "d35415a6-2fc4-4544-8b27-cd22e3a87b00") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:16.924879 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:16.924838    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs\") pod \"network-metrics-daemon-6psxt\" (UID: \"dac4f70d-7789-4a02-8c3a-e2f28245e0ec\") " pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:16.925064 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:16.925003    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:16.925111 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:16.925067    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs podName:dac4f70d-7789-4a02-8c3a-e2f28245e0ec nodeName:}" failed. No retries permitted until 2026-09-18 16:05:24.925049788 +0000 UTC m=+19.872548552 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs") pod "network-metrics-daemon-6psxt" (UID: "dac4f70d-7789-4a02-8c3a-e2f28245e0ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:17.025585 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:17.025502    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnd69\" (UniqueName: \"kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69\") pod \"network-check-target-j5sx5\" (UID: \"e94bcc01-a5f8-49ef-ac80-07170e29c00d\") " pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:17.025769 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:17.025698    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:05:17.025769 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:17.025726    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:05:17.025769 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:17.025736    2554 projected.go:196] Error preparing data for projected volume kube-api-access-xnd69 for pod openshift-network-diagnostics/network-check-target-j5sx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:17.025893 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:17.025787    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69 podName:e94bcc01-a5f8-49ef-ac80-07170e29c00d nodeName:}" failed. No retries permitted until 2026-09-18 16:05:25.025773647 +0000 UTC m=+19.973272402 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xnd69" (UniqueName: "kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69") pod "network-check-target-j5sx5" (UID: "e94bcc01-a5f8-49ef-ac80-07170e29c00d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:17.272546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:17.272452    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:17.272677 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:17.272602    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6psxt" podUID="dac4f70d-7789-4a02-8c3a-e2f28245e0ec"
Sep 18 16:05:17.273024 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:17.273007    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:17.273119 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:17.273099    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5sx5" podUID="e94bcc01-a5f8-49ef-ac80-07170e29c00d"
Sep 18 16:05:17.273286 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:17.273264    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:17.273393 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:17.273371    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c8d4n" podUID="d35415a6-2fc4-4544-8b27-cd22e3a87b00"
Sep 18 16:05:19.271912 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:19.271838    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:19.271912 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:19.271871    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:19.272497 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:19.271838    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:19.272497 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:19.271976    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5sx5" podUID="e94bcc01-a5f8-49ef-ac80-07170e29c00d"
Sep 18 16:05:19.272497 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:19.272081    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6psxt" podUID="dac4f70d-7789-4a02-8c3a-e2f28245e0ec"
Sep 18 16:05:19.272497 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:19.272220    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c8d4n" podUID="d35415a6-2fc4-4544-8b27-cd22e3a87b00"
Sep 18 16:05:21.274065 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:21.274031    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:21.274506 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:21.274121    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:21.274506 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:21.274124    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5sx5" podUID="e94bcc01-a5f8-49ef-ac80-07170e29c00d"
Sep 18 16:05:21.274506 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:21.274250    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c8d4n" podUID="d35415a6-2fc4-4544-8b27-cd22e3a87b00"
Sep 18 16:05:21.274506 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:21.274282    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:21.274506 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:21.274403    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6psxt" podUID="dac4f70d-7789-4a02-8c3a-e2f28245e0ec"
Sep 18 16:05:23.274704 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:23.274673    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:23.275166 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:23.274673    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:23.275166 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:23.274775    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c8d4n" podUID="d35415a6-2fc4-4544-8b27-cd22e3a87b00"
Sep 18 16:05:23.275166 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:23.274675    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:23.275166 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:23.274860    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5sx5" podUID="e94bcc01-a5f8-49ef-ac80-07170e29c00d"
Sep 18 16:05:23.275166 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:23.274941    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6psxt" podUID="dac4f70d-7789-4a02-8c3a-e2f28245e0ec"
Sep 18 16:05:24.889999 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:24.889960    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret\") pod \"global-pull-secret-syncer-c8d4n\" (UID: \"d35415a6-2fc4-4544-8b27-cd22e3a87b00\") " pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:24.890463 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:24.890132    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:24.890463 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:24.890210    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret podName:d35415a6-2fc4-4544-8b27-cd22e3a87b00 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:40.890189235 +0000 UTC m=+35.837687987 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret") pod "global-pull-secret-syncer-c8d4n" (UID: "d35415a6-2fc4-4544-8b27-cd22e3a87b00") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:24.990352 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:24.990318    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs\") pod \"network-metrics-daemon-6psxt\" (UID: \"dac4f70d-7789-4a02-8c3a-e2f28245e0ec\") " pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:24.990577 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:24.990442    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:24.990577 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:24.990509    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs podName:dac4f70d-7789-4a02-8c3a-e2f28245e0ec nodeName:}" failed. No retries permitted until 2026-09-18 16:05:40.990488601 +0000 UTC m=+35.937987355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs") pod "network-metrics-daemon-6psxt" (UID: "dac4f70d-7789-4a02-8c3a-e2f28245e0ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:25.090623 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:25.090593    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnd69\" (UniqueName: \"kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69\") pod \"network-check-target-j5sx5\" (UID: \"e94bcc01-a5f8-49ef-ac80-07170e29c00d\") " pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:25.090787 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:25.090769    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:05:25.090835 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:25.090791    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:05:25.090835 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:25.090800    2554 projected.go:196] Error preparing data for projected volume kube-api-access-xnd69 for pod openshift-network-diagnostics/network-check-target-j5sx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:25.090913 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:25.090854    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69 podName:e94bcc01-a5f8-49ef-ac80-07170e29c00d nodeName:}" failed. No retries permitted until 2026-09-18 16:05:41.090839311 +0000 UTC m=+36.038338067 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xnd69" (UniqueName: "kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69") pod "network-check-target-j5sx5" (UID: "e94bcc01-a5f8-49ef-ac80-07170e29c00d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:25.271330 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:25.271250    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:25.271464 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:25.271248    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:25.271464 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:25.271389    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c8d4n" podUID="d35415a6-2fc4-4544-8b27-cd22e3a87b00"
Sep 18 16:05:25.271566 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:25.271475    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5sx5" podUID="e94bcc01-a5f8-49ef-ac80-07170e29c00d"
Sep 18 16:05:25.271566 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:25.271250    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:25.271686 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:25.271597    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6psxt" podUID="dac4f70d-7789-4a02-8c3a-e2f28245e0ec"
Sep 18 16:05:27.272392 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:27.272349    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:27.272892 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:27.272473    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c8d4n" podUID="d35415a6-2fc4-4544-8b27-cd22e3a87b00"
Sep 18 16:05:27.272892 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:27.272507    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:27.272892 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:27.272674    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6psxt" podUID="dac4f70d-7789-4a02-8c3a-e2f28245e0ec"
Sep 18 16:05:27.272892 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:27.272719    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:27.272892 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:27.272828    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5sx5" podUID="e94bcc01-a5f8-49ef-ac80-07170e29c00d"
Sep 18 16:05:28.454006 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.453771    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-26zfw" event={"ID":"4d012abb-e750-4643-b3fe-8cffd9723e76","Type":"ContainerStarted","Data":"9fd6d265b69c434c0c7fc99e4bcd024f2ccdbe3770f49cf85acd7bda9ae65626"}
Sep 18 16:05:28.455341 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.455317    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj" event={"ID":"c6df82d1-c147-481e-87bc-18da5165d85f","Type":"ContainerStarted","Data":"36666dbba02ce5ef598ead00a8874b48659c3cfb258553d5cf43dd126b063c78"}
Sep 18 16:05:28.456607 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.456585    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9286k" event={"ID":"8906b592-b7c8-46a7-ab14-c89b0803fd8d","Type":"ContainerStarted","Data":"12436df9006d895944d9e7a3a4651ef4b23ac1c81475a1c62fca2b378da9cdaf"}
Sep 18 16:05:28.458166 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.458144    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w26s7" event={"ID":"60f8bc85-96e6-4009-808d-914efbfd97e1","Type":"ContainerStarted","Data":"491b58457813c15d047e035e36e690b65fb2159436415ae383b430fe18515ce9"}
Sep 18 16:05:28.459675 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.459653    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vzt5f" event={"ID":"cf58bf07-5e75-4a2c-9168-29781769dec1","Type":"ContainerStarted","Data":"f7bcb601f80b90e487938df76a2975ff6c11b14b1c850c33a2d067e49b1279b4"}
Sep 18 16:05:28.461984 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.461965    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb" event={"ID":"e7675747-b9b8-4eac-b41f-5eb3bf62d074","Type":"ContainerStarted","Data":"38179cdf6f56501245d6d948b7f322891ad97e8b4004b218e7ed3ac49d480900"}
Sep 18 16:05:28.462053 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.462016    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb" event={"ID":"e7675747-b9b8-4eac-b41f-5eb3bf62d074","Type":"ContainerStarted","Data":"73d2676d4c5f9abd2a77d4b68905acfad1ae8bd1d13350a8329a0aca62b3b2cb"}
Sep 18 16:05:28.462098 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.462060    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb" event={"ID":"e7675747-b9b8-4eac-b41f-5eb3bf62d074","Type":"ContainerStarted","Data":"c2bcf548186b9f435b45cd0174a6b98439fc8498dcf6f252c63dda457f3e7a03"}
Sep 18 16:05:28.462098 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.462091    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb" event={"ID":"e7675747-b9b8-4eac-b41f-5eb3bf62d074","Type":"ContainerStarted","Data":"741276ac65a1d9e1187cd273f9b3fd9be56a41351cdc6cc8b7886ac9d32ccbb0"}
Sep 18 16:05:28.463513 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.463489    2554 generic.go:356] "Generic (PLEG): container finished" podID="1c78da59-56bd-4f33-a7e5-21f3f39e0446" containerID="3f5147214d5542bce79dae8877173c581e85e7ac417b26e2e451b72dd8a012b1" exitCode=0
Sep 18 16:05:28.463617 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.463556    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nw2sh" event={"ID":"1c78da59-56bd-4f33-a7e5-21f3f39e0446","Type":"ContainerDied","Data":"3f5147214d5542bce79dae8877173c581e85e7ac417b26e2e451b72dd8a012b1"}
Sep 18 16:05:28.465142 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.465125    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wwwrj" event={"ID":"ae7e13d9-6ffe-47ec-8e85-a19bd587f9ed","Type":"ContainerStarted","Data":"2a587822951325619472013df1b37f37bdec6bc4240e246f2256e248ab3e2991"}
Sep 18 16:05:28.472780 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.472739    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-26zfw" podStartSLOduration=3.829103689 podStartE2EDuration="21.472728362s" totalImagesPullingTime="17.643624673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:28.472658264 +0000 UTC m=+23.420157039" watchObservedRunningTime="2026-09-18 16:05:28.472728362 +0000 UTC m=+23.420227132"
Sep 18 16:05:28.518601 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.518516    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wwwrj" podStartSLOduration=3.876899976 podStartE2EDuration="21.51849482s" totalImagesPullingTime="17.641594844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:28.518464477 +0000 UTC m=+23.465963254" watchObservedRunningTime="2026-09-18 16:05:28.51849482 +0000 UTC m=+23.465993595"
Sep 18 16:05:28.535101 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.535049    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vzt5f" podStartSLOduration=3.90754455 podStartE2EDuration="21.535029421s" totalImagesPullingTime="17.627484871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:28.534429844 +0000 UTC m=+23.481928620" watchObservedRunningTime="2026-09-18 16:05:28.535029421 +0000 UTC m=+23.482528196"
Sep 18 16:05:28.570892 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.570821    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9286k" podStartSLOduration=3.922725242 podStartE2EDuration="21.570798883s" totalImagesPullingTime="17.648073641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:28.552024977 +0000 UTC m=+23.499523752" watchObservedRunningTime="2026-09-18 16:05:28.570798883 +0000 UTC m=+23.518297673"
Sep 18 16:05:28.571039 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:28.570975    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w26s7" podStartSLOduration=3.880305341 podStartE2EDuration="21.570962366s" totalImagesPullingTime="17.690657025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:28.569852528 +0000 UTC m=+23.517351305" watchObservedRunningTime="2026-09-18 16:05:28.570962366 +0000 UTC m=+23.518461142"
Sep 18 16:05:29.012571 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:29.012513    2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 16:05:29.229197 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:29.229024    2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T16:05:29.012571077Z","UUID":"04874316-0235-4b73-a3e5-2181224bb840","Handler":null,"Name":"","Endpoint":""}
Sep 18 16:05:29.230795 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:29.230765    2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 16:05:29.230918 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:29.230804    2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 16:05:29.271389 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:29.271358    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:29.271514 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:29.271363    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:29.271605 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:29.271460    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c8d4n" podUID="d35415a6-2fc4-4544-8b27-cd22e3a87b00"
Sep 18 16:05:29.271605 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:29.271585    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6psxt" podUID="dac4f70d-7789-4a02-8c3a-e2f28245e0ec"
Sep 18 16:05:29.271723 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:29.271650    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:29.271773 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:29.271728    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5sx5" podUID="e94bcc01-a5f8-49ef-ac80-07170e29c00d"
Sep 18 16:05:29.469334 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:29.469298    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f77cv" event={"ID":"1c5ec270-9dc8-45c4-bfb5-90f150a2fe54","Type":"ContainerStarted","Data":"e245026093ddda67d121f9c15d3e0fd0752ed8a4e84d7d999d2ade6aea63dba8"}
Sep 18 16:05:29.473904 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:29.473875    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb" event={"ID":"e7675747-b9b8-4eac-b41f-5eb3bf62d074","Type":"ContainerStarted","Data":"be393feb3d0668f7602d587b3e81efb067ba011c4352842519f8d6975f3e5a8a"}
Sep 18 16:05:29.474034 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:29.473955    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb" event={"ID":"e7675747-b9b8-4eac-b41f-5eb3bf62d074","Type":"ContainerStarted","Data":"76381a3b388e92cf0585b609e2a16e8ad08f496d3fa8c4916a17c4fe934cdd9d"}
Sep 18 16:05:29.475877 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:29.475795    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj" event={"ID":"c6df82d1-c147-481e-87bc-18da5165d85f","Type":"ContainerStarted","Data":"0a882f1553f1e4fa6429936bbd23cf7ebd28d63c20b111bc21b3af8b53f316c8"}
Sep 18 16:05:30.481136 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:30.481096    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj" event={"ID":"c6df82d1-c147-481e-87bc-18da5165d85f","Type":"ContainerStarted","Data":"999cebe4c0d3b2e21eb9d0d5e87535f109f110dec997fcc8dda536d871703809"}
Sep 18 16:05:30.499023 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:30.498969    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65fqj" podStartSLOduration=4.290984492 podStartE2EDuration="23.498949303s" totalImagesPullingTime="19.207964811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:30.498364912 +0000 UTC m=+25.445863685" watchObservedRunningTime="2026-09-18 16:05:30.498949303 +0000 UTC m=+25.446448078"
Sep 18 16:05:30.499266 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:30.499246    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f77cv" podStartSLOduration=5.870775614 podStartE2EDuration="23.499239487s" totalImagesPullingTime="17.628463873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:29.48838578 +0000 UTC m=+24.435884555" watchObservedRunningTime="2026-09-18 16:05:30.499239487 +0000 UTC m=+25.446738262"
Sep 18 16:05:31.274877 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:31.274802    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:31.275052 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:31.274924    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:31.275052 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:31.274932    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c8d4n" podUID="d35415a6-2fc4-4544-8b27-cd22e3a87b00"
Sep 18 16:05:31.275052 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:31.274957    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:31.275221 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:31.275078    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6psxt" podUID="dac4f70d-7789-4a02-8c3a-e2f28245e0ec"
Sep 18 16:05:31.275221 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:31.275187    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5sx5" podUID="e94bcc01-a5f8-49ef-ac80-07170e29c00d"
Sep 18 16:05:31.487118 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:31.487083    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb" event={"ID":"e7675747-b9b8-4eac-b41f-5eb3bf62d074","Type":"ContainerStarted","Data":"df2d81bfba700dbd51544d54ec56638bbfbccdd57e26e149a135d5bead34416a"}
Sep 18 16:05:32.623845 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:32.623813    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vzt5f"
Sep 18 16:05:32.624320 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:32.624301    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vzt5f"
Sep 18 16:05:33.272122 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:33.271902    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:33.272275 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:33.271905    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:33.272275 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:33.272154    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5sx5" podUID="e94bcc01-a5f8-49ef-ac80-07170e29c00d"
Sep 18 16:05:33.272275 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:33.271965    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:33.272275 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:33.272225    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c8d4n" podUID="d35415a6-2fc4-4544-8b27-cd22e3a87b00"
Sep 18 16:05:33.272450 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:33.272331    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6psxt" podUID="dac4f70d-7789-4a02-8c3a-e2f28245e0ec"
Sep 18 16:05:33.495122 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:33.495090    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb" event={"ID":"e7675747-b9b8-4eac-b41f-5eb3bf62d074","Type":"ContainerStarted","Data":"8ff5d11ebac6a6fc0d6b8fc48f898bdc7dae1560dd5f488ef22765894618eaae"}
Sep 18 16:05:33.495364 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:33.495347    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:33.495469 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:33.495422    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:33.497155 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:33.497132    2554 generic.go:356] "Generic (PLEG): container finished" podID="1c78da59-56bd-4f33-a7e5-21f3f39e0446" containerID="7b16046799102734ff9b3305c5b1b59fbcebe32587d2ff0b8e3b94391bf1a896" exitCode=0
Sep 18 16:05:33.497241 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:33.497208    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nw2sh" event={"ID":"1c78da59-56bd-4f33-a7e5-21f3f39e0446","Type":"ContainerDied","Data":"7b16046799102734ff9b3305c5b1b59fbcebe32587d2ff0b8e3b94391bf1a896"}
Sep 18 16:05:33.497453 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:33.497436    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vzt5f"
Sep 18 16:05:33.497935 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:33.497879    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vzt5f"
Sep 18 16:05:33.511485 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:33.511464    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:33.521676 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:33.521639    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb" podStartSLOduration=8.672724508 podStartE2EDuration="26.521626387s" totalImagesPullingTime="17.848901879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:33.521215184 +0000 UTC m=+28.468713959" watchObservedRunningTime="2026-09-18 16:05:33.521626387 +0000 UTC m=+28.469125161"
Sep 18 16:05:34.500072 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:34.500043    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:34.514290 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:34.514260    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:05:35.178126 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:35.177839    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j5sx5"]
Sep 18 16:05:35.178283 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:35.178143    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:35.178283 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:35.178231    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5sx5" podUID="e94bcc01-a5f8-49ef-ac80-07170e29c00d"
Sep 18 16:05:35.180887 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:35.180862    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6psxt"]
Sep 18 16:05:35.181034 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:35.180963    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:35.181096 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:35.181059    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6psxt" podUID="dac4f70d-7789-4a02-8c3a-e2f28245e0ec"
Sep 18 16:05:35.181504 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:35.181485    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c8d4n"]
Sep 18 16:05:35.181636 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:35.181623    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:35.181744 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:35.181722    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c8d4n" podUID="d35415a6-2fc4-4544-8b27-cd22e3a87b00"
Sep 18 16:05:35.503410 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:35.503331    2554 generic.go:356] "Generic (PLEG): container finished" podID="1c78da59-56bd-4f33-a7e5-21f3f39e0446" containerID="d8163ef65edfd917bc518b0de9c2c0e9095501b5f8d8ad2fd3141fd23a7e12a2" exitCode=0
Sep 18 16:05:35.503410 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:35.503381    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nw2sh" event={"ID":"1c78da59-56bd-4f33-a7e5-21f3f39e0446","Type":"ContainerDied","Data":"d8163ef65edfd917bc518b0de9c2c0e9095501b5f8d8ad2fd3141fd23a7e12a2"}
Sep 18 16:05:37.274209 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:37.274139    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:37.274642 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:37.274208    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:37.274642 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:37.274224    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5sx5" podUID="e94bcc01-a5f8-49ef-ac80-07170e29c00d"
Sep 18 16:05:37.274642 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:37.274318    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c8d4n" podUID="d35415a6-2fc4-4544-8b27-cd22e3a87b00"
Sep 18 16:05:37.274642 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:37.274384    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:37.274642 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:37.274473    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6psxt" podUID="dac4f70d-7789-4a02-8c3a-e2f28245e0ec"
Sep 18 16:05:37.509436 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:37.509407    2554 generic.go:356] "Generic (PLEG): container finished" podID="1c78da59-56bd-4f33-a7e5-21f3f39e0446" containerID="1756a4886b47aa314ee93f260f287fdef461d66c699bfcc23161fcce0ac196d6" exitCode=0
Sep 18 16:05:37.509604 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:37.509452    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nw2sh" event={"ID":"1c78da59-56bd-4f33-a7e5-21f3f39e0446","Type":"ContainerDied","Data":"1756a4886b47aa314ee93f260f287fdef461d66c699bfcc23161fcce0ac196d6"}
Sep 18 16:05:39.274457 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:39.274422    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:39.274908 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:39.274546    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:39.274908 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:39.274570    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:39.274908 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:39.274565    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5sx5" podUID="e94bcc01-a5f8-49ef-ac80-07170e29c00d"
Sep 18 16:05:39.274908 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:39.274642    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c8d4n" podUID="d35415a6-2fc4-4544-8b27-cd22e3a87b00"
Sep 18 16:05:39.274908 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:39.274727    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6psxt" podUID="dac4f70d-7789-4a02-8c3a-e2f28245e0ec"
Sep 18 16:05:40.913184 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:40.913109    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret\") pod \"global-pull-secret-syncer-c8d4n\" (UID: \"d35415a6-2fc4-4544-8b27-cd22e3a87b00\") " pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:40.913594 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:40.913289    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:40.913594 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:40.913370    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret podName:d35415a6-2fc4-4544-8b27-cd22e3a87b00 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:12.913349177 +0000 UTC m=+67.860847945 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret") pod "global-pull-secret-syncer-c8d4n" (UID: "d35415a6-2fc4-4544-8b27-cd22e3a87b00") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:41.014533 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.014485    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs\") pod \"network-metrics-daemon-6psxt\" (UID: \"dac4f70d-7789-4a02-8c3a-e2f28245e0ec\") " pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:41.014704 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:41.014643    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:41.014767 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:41.014714    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs podName:dac4f70d-7789-4a02-8c3a-e2f28245e0ec nodeName:}" failed. No retries permitted until 2026-09-18 16:06:13.014699193 +0000 UTC m=+67.962197949 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs") pod "network-metrics-daemon-6psxt" (UID: "dac4f70d-7789-4a02-8c3a-e2f28245e0ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:41.023483 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.023460    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-195.ec2.internal" event="NodeReady"
Sep 18 16:05:41.023652 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.023637    2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 16:05:41.064694 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.064666    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xxw4k"]
Sep 18 16:05:41.092594 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.092566    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gsxcc"]
Sep 18 16:05:41.092733 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.092709    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:41.095027 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.095005    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 16:05:41.095137 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.095005    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bgqnj\""
Sep 18 16:05:41.095137 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.095004    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 16:05:41.108633 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.108612    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xxw4k"]
Sep 18 16:05:41.108731 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.108646    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gsxcc"]
Sep 18 16:05:41.108787 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.108730    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gsxcc"
Sep 18 16:05:41.110794 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.110773    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 16:05:41.110899 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.110840    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 16:05:41.110899 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.110850    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-69r9k\""
Sep 18 16:05:41.110899 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.110781    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 16:05:41.114926 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.114905    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnd69\" (UniqueName: \"kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69\") pod \"network-check-target-j5sx5\" (UID: \"e94bcc01-a5f8-49ef-ac80-07170e29c00d\") " pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:41.115078 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:41.115063    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:05:41.115133 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:41.115084    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:05:41.115133 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:41.115093    2554 projected.go:196] Error preparing data for projected volume kube-api-access-xnd69 for pod openshift-network-diagnostics/network-check-target-j5sx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:41.115198 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:41.115144    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69 podName:e94bcc01-a5f8-49ef-ac80-07170e29c00d nodeName:}" failed. No retries permitted until 2026-09-18 16:06:13.115132504 +0000 UTC m=+68.062631259 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xnd69" (UniqueName: "kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69") pod "network-check-target-j5sx5" (UID: "e94bcc01-a5f8-49ef-ac80-07170e29c00d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:41.215615 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.215578    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lmbp\" (UniqueName: \"kubernetes.io/projected/cca8c0d4-a36f-4622-8f97-e18b71343f72-kube-api-access-6lmbp\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:41.215806 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.215646    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09f70d2c-590d-4474-ac55-e55056a054cd-cert\") pod \"ingress-canary-gsxcc\" (UID: \"09f70d2c-590d-4474-ac55-e55056a054cd\") " pod="openshift-ingress-canary/ingress-canary-gsxcc"
Sep 18 16:05:41.215806 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.215678    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:41.215806 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.215700    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cca8c0d4-a36f-4622-8f97-e18b71343f72-config-volume\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:41.215806 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.215744    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4klq\" (UniqueName: \"kubernetes.io/projected/09f70d2c-590d-4474-ac55-e55056a054cd-kube-api-access-g4klq\") pod \"ingress-canary-gsxcc\" (UID: \"09f70d2c-590d-4474-ac55-e55056a054cd\") " pod="openshift-ingress-canary/ingress-canary-gsxcc"
Sep 18 16:05:41.215984 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.215824    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca8c0d4-a36f-4622-8f97-e18b71343f72-tmp-dir\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:41.270904 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.270871    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:05:41.271224 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.270872    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:05:41.271618 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.270872    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:05:41.273446 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.273307    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:05:41.273446 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.273339    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 16:05:41.273446 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.273393    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 16:05:41.273446 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.273433    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j9g75\""
Sep 18 16:05:41.273762 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.273741    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:05:41.273839 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.273769    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89djg\""
Sep 18 16:05:41.316784 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.316755    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lmbp\" (UniqueName: \"kubernetes.io/projected/cca8c0d4-a36f-4622-8f97-e18b71343f72-kube-api-access-6lmbp\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:41.316907 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.316853    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09f70d2c-590d-4474-ac55-e55056a054cd-cert\") pod \"ingress-canary-gsxcc\" (UID: \"09f70d2c-590d-4474-ac55-e55056a054cd\") " pod="openshift-ingress-canary/ingress-canary-gsxcc"
Sep 18 16:05:41.316907 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.316887    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:41.317021 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.316916    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cca8c0d4-a36f-4622-8f97-e18b71343f72-config-volume\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:41.317021 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.316953    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4klq\" (UniqueName: \"kubernetes.io/projected/09f70d2c-590d-4474-ac55-e55056a054cd-kube-api-access-g4klq\") pod \"ingress-canary-gsxcc\" (UID: \"09f70d2c-590d-4474-ac55-e55056a054cd\") " pod="openshift-ingress-canary/ingress-canary-gsxcc"
Sep 18 16:05:41.317021 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.316992    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca8c0d4-a36f-4622-8f97-e18b71343f72-tmp-dir\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:41.317021 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:41.317001    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:05:41.317236 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:41.317062    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls podName:cca8c0d4-a36f-4622-8f97-e18b71343f72 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:41.817043889 +0000 UTC m=+36.764542641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls") pod "dns-default-xxw4k" (UID: "cca8c0d4-a36f-4622-8f97-e18b71343f72") : secret "dns-default-metrics-tls" not found
Sep 18 16:05:41.317342 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.317314    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca8c0d4-a36f-4622-8f97-e18b71343f72-tmp-dir\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:41.317478 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.317458    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cca8c0d4-a36f-4622-8f97-e18b71343f72-config-volume\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:41.321946 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.321797    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09f70d2c-590d-4474-ac55-e55056a054cd-cert\") pod \"ingress-canary-gsxcc\" (UID: \"09f70d2c-590d-4474-ac55-e55056a054cd\") " pod="openshift-ingress-canary/ingress-canary-gsxcc"
Sep 18 16:05:41.324950 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.324916    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4klq\" (UniqueName: \"kubernetes.io/projected/09f70d2c-590d-4474-ac55-e55056a054cd-kube-api-access-g4klq\") pod \"ingress-canary-gsxcc\" (UID: \"09f70d2c-590d-4474-ac55-e55056a054cd\") " pod="openshift-ingress-canary/ingress-canary-gsxcc"
Sep 18 16:05:41.325049 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.325012    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lmbp\" (UniqueName: \"kubernetes.io/projected/cca8c0d4-a36f-4622-8f97-e18b71343f72-kube-api-access-6lmbp\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:41.419634 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.419593    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gsxcc"
Sep 18 16:05:41.820730 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.820707    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:41.820842 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:41.820820    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:05:41.820902 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:41.820874    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls podName:cca8c0d4-a36f-4622-8f97-e18b71343f72 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:42.82085802 +0000 UTC m=+37.768356787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls") pod "dns-default-xxw4k" (UID: "cca8c0d4-a36f-4622-8f97-e18b71343f72") : secret "dns-default-metrics-tls" not found
Sep 18 16:05:41.842732 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:41.842707    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gsxcc"]
Sep 18 16:05:41.848418 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:05:41.848393    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f70d2c_590d_4474_ac55_e55056a054cd.slice/crio-771c07bd1adadc40fd907d6792db378c8b58e118dfdcc4c2ff4e598895fb3e48 WatchSource:0}: Error finding container 771c07bd1adadc40fd907d6792db378c8b58e118dfdcc4c2ff4e598895fb3e48: Status 404 returned error can't find the container with id 771c07bd1adadc40fd907d6792db378c8b58e118dfdcc4c2ff4e598895fb3e48
Sep 18 16:05:42.524422 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:42.524384    2554 generic.go:356] "Generic (PLEG): container finished" podID="1c78da59-56bd-4f33-a7e5-21f3f39e0446" containerID="a870830b3f8690356b6959db114e9c533a53c145c5cd2042bb223f1f836b6ef5" exitCode=0
Sep 18 16:05:42.525201 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:42.524455    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nw2sh" event={"ID":"1c78da59-56bd-4f33-a7e5-21f3f39e0446","Type":"ContainerDied","Data":"a870830b3f8690356b6959db114e9c533a53c145c5cd2042bb223f1f836b6ef5"}
Sep 18 16:05:42.527631 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:42.527598    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gsxcc" event={"ID":"09f70d2c-590d-4474-ac55-e55056a054cd","Type":"ContainerStarted","Data":"771c07bd1adadc40fd907d6792db378c8b58e118dfdcc4c2ff4e598895fb3e48"}
Sep 18 16:05:42.830100 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:42.830070    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:42.830396 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:42.830376    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:05:42.830464 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:42.830453    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls podName:cca8c0d4-a36f-4622-8f97-e18b71343f72 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:44.830434776 +0000 UTC m=+39.777933543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls") pod "dns-default-xxw4k" (UID: "cca8c0d4-a36f-4622-8f97-e18b71343f72") : secret "dns-default-metrics-tls" not found
Sep 18 16:05:43.533812 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:43.533777    2554 generic.go:356] "Generic (PLEG): container finished" podID="1c78da59-56bd-4f33-a7e5-21f3f39e0446" containerID="e5e35914cde8b641bce488e8fb95dda8c172771970f538d3c5aa60f3c3065c97" exitCode=0
Sep 18 16:05:43.534271 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:43.533823    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nw2sh" event={"ID":"1c78da59-56bd-4f33-a7e5-21f3f39e0446","Type":"ContainerDied","Data":"e5e35914cde8b641bce488e8fb95dda8c172771970f538d3c5aa60f3c3065c97"}
Sep 18 16:05:44.539431 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:44.539229    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nw2sh" event={"ID":"1c78da59-56bd-4f33-a7e5-21f3f39e0446","Type":"ContainerStarted","Data":"c4719489876503dc843b0bf257aba186d72e9009e94745f4a4a6c0d3b0b454b0"}
Sep 18 16:05:44.540781 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:44.540758    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gsxcc" event={"ID":"09f70d2c-590d-4474-ac55-e55056a054cd","Type":"ContainerStarted","Data":"2a31f9571ca45c42582d2fd88da08ec56933b3dc0f1b08fb3b98ce7cd213cf2a"}
Sep 18 16:05:44.560906 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:44.560860    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nw2sh" podStartSLOduration=9.087811159 podStartE2EDuration="37.56084817s" totalImagesPullingTime="28.473037011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:44.559428352 +0000 UTC m=+39.506927125" watchObservedRunningTime="2026-09-18 16:05:44.56084817 +0000 UTC m=+39.508346944"
Sep 18 16:05:44.574429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:44.574390    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gsxcc" podStartSLOduration=1.189445264 podStartE2EDuration="3.5743772s" totalImagesPullingTime="2.384931936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:44.573984692 +0000 UTC m=+39.521483466" watchObservedRunningTime="2026-09-18 16:05:44.5743772 +0000 UTC m=+39.521875971"
Sep 18 16:05:44.850583 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:44.850547    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:44.850739 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:44.850696    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:05:44.850776 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:44.850753    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls podName:cca8c0d4-a36f-4622-8f97-e18b71343f72 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:48.850740127 +0000 UTC m=+43.798238880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls") pod "dns-default-xxw4k" (UID: "cca8c0d4-a36f-4622-8f97-e18b71343f72") : secret "dns-default-metrics-tls" not found
Sep 18 16:05:48.877100 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:48.877065    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:48.877462 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:48.877184    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:05:48.877462 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:48.877234    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls podName:cca8c0d4-a36f-4622-8f97-e18b71343f72 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:56.877221841 +0000 UTC m=+51.824720592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls") pod "dns-default-xxw4k" (UID: "cca8c0d4-a36f-4622-8f97-e18b71343f72") : secret "dns-default-metrics-tls" not found
Sep 18 16:05:56.925558 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:05:56.925503    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:05:56.925901 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:56.925648    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:05:56.925901 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:05:56.925708    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls podName:cca8c0d4-a36f-4622-8f97-e18b71343f72 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:12.925693196 +0000 UTC m=+67.873191949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls") pod "dns-default-xxw4k" (UID: "cca8c0d4-a36f-4622-8f97-e18b71343f72") : secret "dns-default-metrics-tls" not found
Sep 18 16:06:06.525808 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:06.525776    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f79kb"
Sep 18 16:06:12.931401 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:12.931364    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:06:12.931805 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:12.931425    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret\") pod \"global-pull-secret-syncer-c8d4n\" (UID: \"d35415a6-2fc4-4544-8b27-cd22e3a87b00\") " pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:06:12.931805 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:12.931585    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:06:12.931805 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:12.931665    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls podName:cca8c0d4-a36f-4622-8f97-e18b71343f72 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:44.931648315 +0000 UTC m=+99.879147088 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls") pod "dns-default-xxw4k" (UID: "cca8c0d4-a36f-4622-8f97-e18b71343f72") : secret "dns-default-metrics-tls" not found
Sep 18 16:06:12.934543 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:12.934511    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:06:12.945250 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:12.945229    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d35415a6-2fc4-4544-8b27-cd22e3a87b00-original-pull-secret\") pod \"global-pull-secret-syncer-c8d4n\" (UID: \"d35415a6-2fc4-4544-8b27-cd22e3a87b00\") " pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:06:13.032593 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:13.032554    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs\") pod \"network-metrics-daemon-6psxt\" (UID: \"dac4f70d-7789-4a02-8c3a-e2f28245e0ec\") " pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:06:13.035107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:13.035090    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:06:13.042921 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:13.042903    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 16:06:13.042984 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:13.042969    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs podName:dac4f70d-7789-4a02-8c3a-e2f28245e0ec nodeName:}" failed. No retries permitted until 2026-09-18 16:07:17.042952908 +0000 UTC m=+131.990451675 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs") pod "network-metrics-daemon-6psxt" (UID: "dac4f70d-7789-4a02-8c3a-e2f28245e0ec") : secret "metrics-daemon-secret" not found
Sep 18 16:06:13.098582 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:13.098550    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c8d4n"
Sep 18 16:06:13.133094 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:13.133068    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnd69\" (UniqueName: \"kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69\") pod \"network-check-target-j5sx5\" (UID: \"e94bcc01-a5f8-49ef-ac80-07170e29c00d\") " pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:06:13.136182 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:13.136156    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 16:06:13.146934 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:13.146896    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 16:06:13.157891 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:13.157832    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnd69\" (UniqueName: \"kubernetes.io/projected/e94bcc01-a5f8-49ef-ac80-07170e29c00d-kube-api-access-xnd69\") pod \"network-check-target-j5sx5\" (UID: \"e94bcc01-a5f8-49ef-ac80-07170e29c00d\") " pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:06:13.365823 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:13.365753    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c8d4n"]
Sep 18 16:06:13.369218 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:06:13.369194    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35415a6_2fc4_4544_8b27_cd22e3a87b00.slice/crio-756586ed07a5a489bff1cbfe290d82ad76f0ecac666542eef07446d5f4799ce6 WatchSource:0}: Error finding container 756586ed07a5a489bff1cbfe290d82ad76f0ecac666542eef07446d5f4799ce6: Status 404 returned error can't find the container with id 756586ed07a5a489bff1cbfe290d82ad76f0ecac666542eef07446d5f4799ce6
Sep 18 16:06:13.395017 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:13.394994    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j9g75\""
Sep 18 16:06:13.403218 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:13.403201    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:06:13.608168 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:13.608136    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c8d4n" event={"ID":"d35415a6-2fc4-4544-8b27-cd22e3a87b00","Type":"ContainerStarted","Data":"756586ed07a5a489bff1cbfe290d82ad76f0ecac666542eef07446d5f4799ce6"}
Sep 18 16:06:13.668692 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:13.668633    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j5sx5"]
Sep 18 16:06:13.673534 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:06:13.673491    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94bcc01_a5f8_49ef_ac80_07170e29c00d.slice/crio-19c1e9b5b07d0e538db94fac510ec603fa1f9ade0a189ab664d12167c6eda0d7 WatchSource:0}: Error finding container 19c1e9b5b07d0e538db94fac510ec603fa1f9ade0a189ab664d12167c6eda0d7: Status 404 returned error can't find the container with id 19c1e9b5b07d0e538db94fac510ec603fa1f9ade0a189ab664d12167c6eda0d7
Sep 18 16:06:14.613662 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:14.613629    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j5sx5" event={"ID":"e94bcc01-a5f8-49ef-ac80-07170e29c00d","Type":"ContainerStarted","Data":"19c1e9b5b07d0e538db94fac510ec603fa1f9ade0a189ab664d12167c6eda0d7"}
Sep 18 16:06:18.625968 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:18.625868    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c8d4n" event={"ID":"d35415a6-2fc4-4544-8b27-cd22e3a87b00","Type":"ContainerStarted","Data":"d5b545b4e13a2018c33aecf5db8d4a6fe78c5a70c0fddad43b81d76844adf3aa"}
Sep 18 16:06:18.627356 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:18.627334    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j5sx5" event={"ID":"e94bcc01-a5f8-49ef-ac80-07170e29c00d","Type":"ContainerStarted","Data":"de246fe95945771802b979363e4ad3bc3aa0d7a2c2ed035d718f85c41d819617"}
Sep 18 16:06:18.627477 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:18.627453    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:06:18.641823 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:18.641759    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-c8d4n" podStartSLOduration=66.742901421 podStartE2EDuration="1m11.641738481s" totalImagesPullingTime="4.89883706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:06:18.640596694 +0000 UTC m=+73.588095471" watchObservedRunningTime="2026-09-18 16:06:18.641738481 +0000 UTC m=+73.589237256"
Sep 18 16:06:18.655316 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:18.655274    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j5sx5" podStartSLOduration=67.063413933 podStartE2EDuration="1m11.655256524s" totalImagesPullingTime="4.591842591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:06:18.655060269 +0000 UTC m=+73.602559044" watchObservedRunningTime="2026-09-18 16:06:18.655256524 +0000 UTC m=+73.602755301"
Sep 18 16:06:44.951367 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:44.951330    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:06:44.951847 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:44.951491    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:06:44.951847 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:44.951596    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls podName:cca8c0d4-a36f-4622-8f97-e18b71343f72 nodeName:}" failed. No retries permitted until 2026-09-18 16:07:48.951578884 +0000 UTC m=+163.899077640 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls") pod "dns-default-xxw4k" (UID: "cca8c0d4-a36f-4622-8f97-e18b71343f72") : secret "dns-default-metrics-tls" not found
Sep 18 16:06:45.552727 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.552697    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k"]
Sep 18 16:06:45.555011 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.554993    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k"
Sep 18 16:06:45.557570 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.557550    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:06:45.558395 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.558377    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xhcpl\""
Sep 18 16:06:45.558480 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.558411    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 16:06:45.564880 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.564857    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4"]
Sep 18 16:06:45.566551 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.566514    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k"]
Sep 18 16:06:45.566625 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.566587    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4"
Sep 18 16:06:45.568683 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.568664    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 16:06:45.569446 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.569420    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c2hwb\""
Sep 18 16:06:45.569725 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.569706    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:06:45.569972 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.569957    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 16:06:45.578744 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.578723    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4"]
Sep 18 16:06:45.654730 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.654704    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-9p5b4\" (UID: \"a90376e5-fd27-4200-8823-a836b8027790\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4"
Sep 18 16:06:45.654844 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.654753    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp74t\" (UniqueName: \"kubernetes.io/projected/db1edd1b-c2ec-4a6a-b365-9305e87f8471-kube-api-access-wp74t\") pod \"volume-data-source-validator-5c4db8bdf9-4bh4k\" (UID: \"db1edd1b-c2ec-4a6a-b365-9305e87f8471\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k"
Sep 18 16:06:45.654844 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.654782    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9598p\" (UniqueName: \"kubernetes.io/projected/a90376e5-fd27-4200-8823-a836b8027790-kube-api-access-9598p\") pod \"cluster-samples-operator-685d7d68b5-9p5b4\" (UID: \"a90376e5-fd27-4200-8823-a836b8027790\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4"
Sep 18 16:06:45.660164 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.660140    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp"]
Sep 18 16:06:45.661907 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.661894    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp"
Sep 18 16:06:45.667461 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.667439    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69f947d478-zw294"]
Sep 18 16:06:45.667969 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.667951    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 16:06:45.669369 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.669351    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.670426 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.670409    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:06:45.670676 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.670661    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-m977t\""
Sep 18 16:06:45.671010 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.670996    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 16:06:45.671371 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.671356    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 16:06:45.677451 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.677434    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 16:06:45.682649 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.682622    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bxqxp\""
Sep 18 16:06:45.683658 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.683640    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp"]
Sep 18 16:06:45.684901 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.684882    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 16:06:45.684978 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.684901    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 16:06:45.719273 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.719249    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69f947d478-zw294"]
Sep 18 16:06:45.721212 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.721192    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 16:06:45.755082 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.755058    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8s9n\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-kube-api-access-n8s9n\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.755184 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.755102    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp74t\" (UniqueName: \"kubernetes.io/projected/db1edd1b-c2ec-4a6a-b365-9305e87f8471-kube-api-access-wp74t\") pod \"volume-data-source-validator-5c4db8bdf9-4bh4k\" (UID: \"db1edd1b-c2ec-4a6a-b365-9305e87f8471\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k"
Sep 18 16:06:45.755184 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.755128    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-certificates\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.755184 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.755172    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a5a8e40-2a6b-4289-be46-bf1e50af868e-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-44mkp\" (UID: \"8a5a8e40-2a6b-4289-be46-bf1e50af868e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp"
Sep 18 16:06:45.755401 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.755197    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpjw6\" (UniqueName: \"kubernetes.io/projected/8a5a8e40-2a6b-4289-be46-bf1e50af868e-kube-api-access-vpjw6\") pod \"kube-storage-version-migrator-operator-55889f7b47-44mkp\" (UID: \"8a5a8e40-2a6b-4289-be46-bf1e50af868e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp"
Sep 18 16:06:45.755401 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.755221    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a5a8e40-2a6b-4289-be46-bf1e50af868e-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-44mkp\" (UID: \"8a5a8e40-2a6b-4289-be46-bf1e50af868e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp"
Sep 18 16:06:45.755401 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.755250    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.755401 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.755274    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-installation-pull-secrets\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.755401 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.755304    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9598p\" (UniqueName: \"kubernetes.io/projected/a90376e5-fd27-4200-8823-a836b8027790-kube-api-access-9598p\") pod \"cluster-samples-operator-685d7d68b5-9p5b4\" (UID: \"a90376e5-fd27-4200-8823-a836b8027790\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4"
Sep 18 16:06:45.755401 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.755348    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-trusted-ca\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.755401 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.755387    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-9p5b4\" (UID: \"a90376e5-fd27-4200-8823-a836b8027790\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4"
Sep 18 16:06:45.755401 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.755403    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-image-registry-private-configuration\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.755723 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.755419    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-ca-trust-extracted\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.755723 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.755447    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-bound-sa-token\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.755723 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:45.755479    2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 16:06:45.755723 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:45.755594    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls podName:a90376e5-fd27-4200-8823-a836b8027790 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:46.255573397 +0000 UTC m=+101.203072148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-9p5b4" (UID: "a90376e5-fd27-4200-8823-a836b8027790") : secret "samples-operator-tls" not found
Sep 18 16:06:45.773059 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.773039    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp74t\" (UniqueName: \"kubernetes.io/projected/db1edd1b-c2ec-4a6a-b365-9305e87f8471-kube-api-access-wp74t\") pod \"volume-data-source-validator-5c4db8bdf9-4bh4k\" (UID: \"db1edd1b-c2ec-4a6a-b365-9305e87f8471\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k"
Sep 18 16:06:45.774315 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.774297    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9598p\" (UniqueName: \"kubernetes.io/projected/a90376e5-fd27-4200-8823-a836b8027790-kube-api-access-9598p\") pod \"cluster-samples-operator-685d7d68b5-9p5b4\" (UID: \"a90376e5-fd27-4200-8823-a836b8027790\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4"
Sep 18 16:06:45.856224 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.856152    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-trusted-ca\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.856224 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.856207    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-image-registry-private-configuration\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.856384 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.856242    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-ca-trust-extracted\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.856384 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.856257    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-bound-sa-token\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.856384 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.856278    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8s9n\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-kube-api-access-n8s9n\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.856384 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.856312    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-certificates\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.856384 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.856336    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a5a8e40-2a6b-4289-be46-bf1e50af868e-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-44mkp\" (UID: \"8a5a8e40-2a6b-4289-be46-bf1e50af868e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp"
Sep 18 16:06:45.856669 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.856390    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpjw6\" (UniqueName: \"kubernetes.io/projected/8a5a8e40-2a6b-4289-be46-bf1e50af868e-kube-api-access-vpjw6\") pod \"kube-storage-version-migrator-operator-55889f7b47-44mkp\" (UID: \"8a5a8e40-2a6b-4289-be46-bf1e50af868e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp"
Sep 18 16:06:45.856669 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.856437    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a5a8e40-2a6b-4289-be46-bf1e50af868e-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-44mkp\" (UID: \"8a5a8e40-2a6b-4289-be46-bf1e50af868e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp"
Sep 18 16:06:45.856669 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.856469    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.856669 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.856496    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-installation-pull-secrets\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.856860 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:45.856752    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:06:45.856860 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:45.856767    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f947d478-zw294: secret "image-registry-tls" not found
Sep 18 16:06:45.856860 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:45.856831    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls podName:c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:46.356811926 +0000 UTC m=+101.304310679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls") pod "image-registry-69f947d478-zw294" (UID: "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7") : secret "image-registry-tls" not found
Sep 18 16:06:45.857022 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.856897    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-certificates\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.857022 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.856954    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a5a8e40-2a6b-4289-be46-bf1e50af868e-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-44mkp\" (UID: \"8a5a8e40-2a6b-4289-be46-bf1e50af868e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp"
Sep 18 16:06:45.857022 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.856971    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-ca-trust-extracted\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.857300 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.857220    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-trusted-ca\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.858880 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.858859    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a5a8e40-2a6b-4289-be46-bf1e50af868e-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-44mkp\" (UID: \"8a5a8e40-2a6b-4289-be46-bf1e50af868e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp"
Sep 18 16:06:45.859052 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.859033    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-image-registry-private-configuration\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.859091 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.859070    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-installation-pull-secrets\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.862839 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.862818    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k"
Sep 18 16:06:45.864953 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.864928    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-bound-sa-token\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.865728 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.865635    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpjw6\" (UniqueName: \"kubernetes.io/projected/8a5a8e40-2a6b-4289-be46-bf1e50af868e-kube-api-access-vpjw6\") pod \"kube-storage-version-migrator-operator-55889f7b47-44mkp\" (UID: \"8a5a8e40-2a6b-4289-be46-bf1e50af868e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp"
Sep 18 16:06:45.865979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.865957    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8s9n\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-kube-api-access-n8s9n\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:45.971127 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:45.971084    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp"
Sep 18 16:06:46.129140 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:46.129063    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k"]
Sep 18 16:06:46.132358 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:06:46.132334    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1edd1b_c2ec_4a6a_b365_9305e87f8471.slice/crio-f8575012818d8d7113b92e109deb22e8e7da81589c170d61c810065bb81cc22a WatchSource:0}: Error finding container f8575012818d8d7113b92e109deb22e8e7da81589c170d61c810065bb81cc22a: Status 404 returned error can't find the container with id f8575012818d8d7113b92e109deb22e8e7da81589c170d61c810065bb81cc22a
Sep 18 16:06:46.237823 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:46.237790    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp"]
Sep 18 16:06:46.241104 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:06:46.241077    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5a8e40_2a6b_4289_be46_bf1e50af868e.slice/crio-b650b9aaeb6e4371642a6e63edbfad737d7eedaab9384f60789695d942a72115 WatchSource:0}: Error finding container b650b9aaeb6e4371642a6e63edbfad737d7eedaab9384f60789695d942a72115: Status 404 returned error can't find the container with id b650b9aaeb6e4371642a6e63edbfad737d7eedaab9384f60789695d942a72115
Sep 18 16:06:46.258401 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:46.258374    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-9p5b4\" (UID: \"a90376e5-fd27-4200-8823-a836b8027790\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4"
Sep 18 16:06:46.258557 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:46.258531    2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 16:06:46.258603 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:46.258595    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls podName:a90376e5-fd27-4200-8823-a836b8027790 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:47.25857772 +0000 UTC m=+102.206076476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-9p5b4" (UID: "a90376e5-fd27-4200-8823-a836b8027790") : secret "samples-operator-tls" not found
Sep 18 16:06:46.359310 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:46.359270    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:46.359446 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:46.359385    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:06:46.359446 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:46.359397    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f947d478-zw294: secret "image-registry-tls" not found
Sep 18 16:06:46.359446 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:46.359446    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls podName:c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:47.35943331 +0000 UTC m=+102.306932062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls") pod "image-registry-69f947d478-zw294" (UID: "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7") : secret "image-registry-tls" not found
Sep 18 16:06:46.701611 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:46.701568    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp" event={"ID":"8a5a8e40-2a6b-4289-be46-bf1e50af868e","Type":"ContainerStarted","Data":"b650b9aaeb6e4371642a6e63edbfad737d7eedaab9384f60789695d942a72115"}
Sep 18 16:06:46.702890 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:46.702850    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k" event={"ID":"db1edd1b-c2ec-4a6a-b365-9305e87f8471","Type":"ContainerStarted","Data":"f8575012818d8d7113b92e109deb22e8e7da81589c170d61c810065bb81cc22a"}
Sep 18 16:06:47.266644 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:47.266611    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-9p5b4\" (UID: \"a90376e5-fd27-4200-8823-a836b8027790\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4"
Sep 18 16:06:47.266973 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:47.266754    2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 16:06:47.266973 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:47.266817    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls podName:a90376e5-fd27-4200-8823-a836b8027790 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:49.266801199 +0000 UTC m=+104.214299956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-9p5b4" (UID: "a90376e5-fd27-4200-8823-a836b8027790") : secret "samples-operator-tls" not found
Sep 18 16:06:47.367148 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:47.367113    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:47.367319 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:47.367288    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:06:47.367319 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:47.367308    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f947d478-zw294: secret "image-registry-tls" not found
Sep 18 16:06:47.367397 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:47.367367    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls podName:c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:49.367349872 +0000 UTC m=+104.314848626 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls") pod "image-registry-69f947d478-zw294" (UID: "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7") : secret "image-registry-tls" not found
Sep 18 16:06:47.706770 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:47.706738    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k" event={"ID":"db1edd1b-c2ec-4a6a-b365-9305e87f8471","Type":"ContainerStarted","Data":"5e6aec84b5039c6113276e585d3fd07eadcdfe31f09a41dd4969ba3218c0bc46"}
Sep 18 16:06:47.726250 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:47.726192    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k" podStartSLOduration=1.615485434 podStartE2EDuration="2.726173508s" totalImagesPullingTime="1.110688074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:06:47.725707737 +0000 UTC m=+102.673206511" watchObservedRunningTime="2026-09-18 16:06:47.726173508 +0000 UTC m=+102.673672282"
Sep 18 16:06:49.281086 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:49.281051    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-9p5b4\" (UID: \"a90376e5-fd27-4200-8823-a836b8027790\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4"
Sep 18 16:06:49.281565 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:49.281215    2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 16:06:49.281565 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:49.281281    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls podName:a90376e5-fd27-4200-8823-a836b8027790 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:53.281262155 +0000 UTC m=+108.228760907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-9p5b4" (UID: "a90376e5-fd27-4200-8823-a836b8027790") : secret "samples-operator-tls" not found
Sep 18 16:06:49.381708 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:49.381669    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:49.381870 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:49.381783    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:06:49.381870 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:49.381796    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f947d478-zw294: secret "image-registry-tls" not found
Sep 18 16:06:49.381870 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:49.381845    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls podName:c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:53.381832912 +0000 UTC m=+108.329331664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls") pod "image-registry-69f947d478-zw294" (UID: "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7") : secret "image-registry-tls" not found
Sep 18 16:06:49.632636 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:49.632563    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j5sx5"
Sep 18 16:06:49.716367 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:49.716337    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp" event={"ID":"8a5a8e40-2a6b-4289-be46-bf1e50af868e","Type":"ContainerStarted","Data":"7d4d7a26450a09481d5170ae2708f66657cc9d83bb0cb0c792a89d1468cd61db"}
Sep 18 16:06:49.734719 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:49.734673    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp" podStartSLOduration=1.5845493959999999 podStartE2EDuration="4.734657688s" totalImagesPullingTime="3.150108292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:06:49.733918918 +0000 UTC m=+104.681417694" watchObservedRunningTime="2026-09-18 16:06:49.734657688 +0000 UTC m=+104.682156484"
Sep 18 16:06:50.332393 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:50.332353    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-9nhbr"]
Sep 18 16:06:50.353623 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:50.353590    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-9nhbr"]
Sep 18 16:06:50.353770 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:50.353701    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9nhbr"
Sep 18 16:06:50.356248 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:50.356218    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tww4c\""
Sep 18 16:06:50.356413 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:50.356255    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 16:06:50.356413 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:50.356322    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 16:06:50.490316 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:50.490280    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8qrq\" (UniqueName: \"kubernetes.io/projected/f3cb1491-cb3a-476c-921e-72285a0b3a7d-kube-api-access-q8qrq\") pod \"migrator-76794cbd6c-9nhbr\" (UID: \"f3cb1491-cb3a-476c-921e-72285a0b3a7d\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9nhbr"
Sep 18 16:06:50.590745 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:50.590668    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8qrq\" (UniqueName: \"kubernetes.io/projected/f3cb1491-cb3a-476c-921e-72285a0b3a7d-kube-api-access-q8qrq\") pod \"migrator-76794cbd6c-9nhbr\" (UID: \"f3cb1491-cb3a-476c-921e-72285a0b3a7d\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9nhbr"
Sep 18 16:06:50.599642 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:50.599614    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8qrq\" (UniqueName: \"kubernetes.io/projected/f3cb1491-cb3a-476c-921e-72285a0b3a7d-kube-api-access-q8qrq\") pod \"migrator-76794cbd6c-9nhbr\" (UID: \"f3cb1491-cb3a-476c-921e-72285a0b3a7d\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9nhbr"
Sep 18 16:06:50.662708 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:50.662663    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9nhbr"
Sep 18 16:06:50.978405 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:50.978373    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-9nhbr"]
Sep 18 16:06:50.982046 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:06:50.982012    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cb1491_cb3a_476c_921e_72285a0b3a7d.slice/crio-fb99f81ca34140492915689476331e1e94ba93f5e8eaa8762d0ca9a67daa8346 WatchSource:0}: Error finding container fb99f81ca34140492915689476331e1e94ba93f5e8eaa8762d0ca9a67daa8346: Status 404 returned error can't find the container with id fb99f81ca34140492915689476331e1e94ba93f5e8eaa8762d0ca9a67daa8346
Sep 18 16:06:51.498971 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.498931    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-svv8w"]
Sep 18 16:06:51.524295 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.524259    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-svv8w"]
Sep 18 16:06:51.524460 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.524384    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:51.526691 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.526668    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 16:06:51.526691 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.526680    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 16:06:51.526904 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.526669    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 16:06:51.527478 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.527462    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mnksp\""
Sep 18 16:06:51.527578 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.527464    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 16:06:51.599876 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.599836    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b20bad3b-c233-42b2-8860-2a53b864ade8-crio-socket\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:51.600057 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.599894    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:51.600057 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.599950    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxk9d\" (UniqueName: \"kubernetes.io/projected/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-api-access-xxk9d\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:51.600057 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.599987    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:51.600057 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.600015    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b20bad3b-c233-42b2-8860-2a53b864ade8-data-volume\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:51.700580 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.700542    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:51.700757 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.700585    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxk9d\" (UniqueName: \"kubernetes.io/projected/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-api-access-xxk9d\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:51.700757 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.700614    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:51.700757 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.700638    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b20bad3b-c233-42b2-8860-2a53b864ade8-data-volume\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:51.700757 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:51.700685    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:06:51.700757 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:51.700736    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:51.700757 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.700755    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b20bad3b-c233-42b2-8860-2a53b864ade8-crio-socket\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:51.701083 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:51.700783    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm podName:b20bad3b-c233-42b2-8860-2a53b864ade8 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:52.200761617 +0000 UTC m=+107.148260385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svv8w" (UID: "b20bad3b-c233-42b2-8860-2a53b864ade8") : configmap "kube-rbac-proxy" not found
Sep 18 16:06:51.701083 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:51.700809    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls podName:b20bad3b-c233-42b2-8860-2a53b864ade8 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:52.200799136 +0000 UTC m=+107.148297893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svv8w" (UID: "b20bad3b-c233-42b2-8860-2a53b864ade8") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:51.701083 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.700817    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b20bad3b-c233-42b2-8860-2a53b864ade8-crio-socket\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:51.701083 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.701063    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b20bad3b-c233-42b2-8860-2a53b864ade8-data-volume\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:51.709639 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.709614    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxk9d\" (UniqueName: \"kubernetes.io/projected/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-api-access-xxk9d\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:51.723467 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:51.723438    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9nhbr" event={"ID":"f3cb1491-cb3a-476c-921e-72285a0b3a7d","Type":"ContainerStarted","Data":"fb99f81ca34140492915689476331e1e94ba93f5e8eaa8762d0ca9a67daa8346"}
Sep 18 16:06:52.204578 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:52.204537    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:52.204773 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:52.204590    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:52.204773 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:52.204648    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:06:52.204773 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:52.204722    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm podName:b20bad3b-c233-42b2-8860-2a53b864ade8 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:53.204700021 +0000 UTC m=+108.152198793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svv8w" (UID: "b20bad3b-c233-42b2-8860-2a53b864ade8") : configmap "kube-rbac-proxy" not found
Sep 18 16:06:52.204773 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:52.204754    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:52.204920 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:52.204816    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls podName:b20bad3b-c233-42b2-8860-2a53b864ade8 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:53.204803449 +0000 UTC m=+108.152302204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svv8w" (UID: "b20bad3b-c233-42b2-8860-2a53b864ade8") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:52.730751 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:52.730720    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9nhbr" event={"ID":"f3cb1491-cb3a-476c-921e-72285a0b3a7d","Type":"ContainerStarted","Data":"5ff96a95b4ba65f0ee15e05873b87a58780140909322fce75a646a9c193f1527"}
Sep 18 16:06:53.212821 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:53.212780    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:53.212821 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:53.212820    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:53.213059 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:53.212907    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:06:53.213059 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:53.212939    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:53.213059 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:53.212976    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm podName:b20bad3b-c233-42b2-8860-2a53b864ade8 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:55.212960454 +0000 UTC m=+110.160459206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svv8w" (UID: "b20bad3b-c233-42b2-8860-2a53b864ade8") : configmap "kube-rbac-proxy" not found
Sep 18 16:06:53.213059 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:53.212991    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls podName:b20bad3b-c233-42b2-8860-2a53b864ade8 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:55.212983239 +0000 UTC m=+110.160481991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svv8w" (UID: "b20bad3b-c233-42b2-8860-2a53b864ade8") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:53.313782 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:53.313739    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-9p5b4\" (UID: \"a90376e5-fd27-4200-8823-a836b8027790\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4"
Sep 18 16:06:53.313948 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:53.313887    2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 16:06:53.313988 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:53.313949    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls podName:a90376e5-fd27-4200-8823-a836b8027790 nodeName:}" failed. No retries permitted until 2026-09-18 16:07:01.313930881 +0000 UTC m=+116.261429635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-9p5b4" (UID: "a90376e5-fd27-4200-8823-a836b8027790") : secret "samples-operator-tls" not found
Sep 18 16:06:53.414367 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:53.414328    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:06:53.414553 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:53.414436    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:06:53.414553 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:53.414449    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69f947d478-zw294: secret "image-registry-tls" not found
Sep 18 16:06:53.414553 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:53.414495    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls podName:c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7 nodeName:}" failed. No retries permitted until 2026-09-18 16:07:01.414482089 +0000 UTC m=+116.361980841 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls") pod "image-registry-69f947d478-zw294" (UID: "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7") : secret "image-registry-tls" not found
Sep 18 16:06:53.735635 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:53.735598    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9nhbr" event={"ID":"f3cb1491-cb3a-476c-921e-72285a0b3a7d","Type":"ContainerStarted","Data":"8ff88ca47458c042890c8235975bac4b82dcc01b3f5fc7ac227de00bd17d213e"}
Sep 18 16:06:53.753718 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:53.753669    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9nhbr" podStartSLOduration=2.107246426 podStartE2EDuration="3.753655059s" totalImagesPullingTime="1.646408633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:06:53.752903766 +0000 UTC m=+108.700402540" watchObservedRunningTime="2026-09-18 16:06:53.753655059 +0000 UTC m=+108.701153834"
Sep 18 16:06:54.046083 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:54.045999    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9286k_8906b592-b7c8-46a7-ab14-c89b0803fd8d/dns-node-resolver/0.log"
Sep 18 16:06:55.045032 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:55.045007    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-26zfw_4d012abb-e750-4643-b3fe-8cffd9723e76/node-ca/0.log"
Sep 18 16:06:55.230575 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:55.230537    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:55.230575 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:55.230575    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:55.230798 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:55.230638    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:06:55.230798 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:55.230675    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:55.230798 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:55.230702    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm podName:b20bad3b-c233-42b2-8860-2a53b864ade8 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:59.230685314 +0000 UTC m=+114.178184066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svv8w" (UID: "b20bad3b-c233-42b2-8860-2a53b864ade8") : configmap "kube-rbac-proxy" not found
Sep 18 16:06:55.230798 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:55.230724    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls podName:b20bad3b-c233-42b2-8860-2a53b864ade8 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:59.230712681 +0000 UTC m=+114.178211434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svv8w" (UID: "b20bad3b-c233-42b2-8860-2a53b864ade8") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:55.846174 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:55.846129    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gsxcc_09f70d2c-590d-4474-ac55-e55056a054cd/serve-healthcheck-canary/0.log"
Sep 18 16:06:56.446077 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:56.446048    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-9nhbr_f3cb1491-cb3a-476c-921e-72285a0b3a7d/migrator/0.log"
Sep 18 16:06:56.645917 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:56.645892    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-9nhbr_f3cb1491-cb3a-476c-921e-72285a0b3a7d/graceful-termination/0.log"
Sep 18 16:06:56.846682 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:56.846557    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-44mkp_8a5a8e40-2a6b-4289-be46-bf1e50af868e/kube-storage-version-migrator-operator/0.log"
Sep 18 16:06:59.262882 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:59.262837    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:59.263288 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:06:59.262984    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:06:59.263288 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:59.262994    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:59.263288 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:59.263063    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:06:59.263288 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:59.263072    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls podName:b20bad3b-c233-42b2-8860-2a53b864ade8 nodeName:}" failed. No retries permitted until 2026-09-18 16:07:07.263053047 +0000 UTC m=+122.210551813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svv8w" (UID: "b20bad3b-c233-42b2-8860-2a53b864ade8") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:59.263288 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:06:59.263105    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm podName:b20bad3b-c233-42b2-8860-2a53b864ade8 nodeName:}" failed. No retries permitted until 2026-09-18 16:07:07.26309537 +0000 UTC m=+122.210594128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svv8w" (UID: "b20bad3b-c233-42b2-8860-2a53b864ade8") : configmap "kube-rbac-proxy" not found
Sep 18 16:07:01.380403 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:01.380366    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-9p5b4\" (UID: \"a90376e5-fd27-4200-8823-a836b8027790\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4"
Sep 18 16:07:01.382849 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:01.382831    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a90376e5-fd27-4200-8823-a836b8027790-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-9p5b4\" (UID: \"a90376e5-fd27-4200-8823-a836b8027790\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4"
Sep 18 16:07:01.478960 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:01.478928    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4"
Sep 18 16:07:01.480707 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:01.480691    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:07:01.483037 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:01.483015    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls\") pod \"image-registry-69f947d478-zw294\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") " pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:07:01.577924 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:01.577888    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:07:01.744943 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:01.744825    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4"]
Sep 18 16:07:01.849110 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:01.849088    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69f947d478-zw294"]
Sep 18 16:07:01.851071 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:07:01.851046    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc272d7f1_2c0a_42df_8fbe_e8c8606bbfb7.slice/crio-af316fcc718efccf43a3eebcc0706b208fc750828058f259c4122b0f54c5b5b8 WatchSource:0}: Error finding container af316fcc718efccf43a3eebcc0706b208fc750828058f259c4122b0f54c5b5b8: Status 404 returned error can't find the container with id af316fcc718efccf43a3eebcc0706b208fc750828058f259c4122b0f54c5b5b8
Sep 18 16:07:02.767237 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:02.767194    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f947d478-zw294" event={"ID":"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7","Type":"ContainerStarted","Data":"7d8c2c094521202c613f43551221ba523d029f897cde76c902ee24b8b25691bd"}
Sep 18 16:07:02.767731 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:02.767260    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f947d478-zw294" event={"ID":"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7","Type":"ContainerStarted","Data":"af316fcc718efccf43a3eebcc0706b208fc750828058f259c4122b0f54c5b5b8"}
Sep 18 16:07:02.767731 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:02.767323    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:07:02.769115 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:02.769089    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4" event={"ID":"a90376e5-fd27-4200-8823-a836b8027790","Type":"ContainerStarted","Data":"2998385143f3948d5cdfee334c75a1416efd62410735c1203bc0438e4580dcef"}
Sep 18 16:07:02.786915 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:02.786863    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69f947d478-zw294" podStartSLOduration=17.786850548 podStartE2EDuration="17.786850548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:02.786153889 +0000 UTC m=+117.733652679" watchObservedRunningTime="2026-09-18 16:07:02.786850548 +0000 UTC m=+117.734349319"
Sep 18 16:07:03.774559 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:03.774500    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4" event={"ID":"a90376e5-fd27-4200-8823-a836b8027790","Type":"ContainerStarted","Data":"3390e4c72b8e5d8b0c44631630ccb17b66de9e2422fce5c66a08b8136499b8fa"}
Sep 18 16:07:03.774936 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:03.774604    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4" event={"ID":"a90376e5-fd27-4200-8823-a836b8027790","Type":"ContainerStarted","Data":"8b4676e8c7fd470b4ca14bb88b13117c11bb9f23b4d30dac01416347fdc87d72"}
Sep 18 16:07:03.789185 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:03.789126    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-9p5b4" podStartSLOduration=16.955104015 podStartE2EDuration="18.789108441s" totalImagesPullingTime="1.834004426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:03.788656745 +0000 UTC m=+118.736155520" watchObservedRunningTime="2026-09-18 16:07:03.789108441 +0000 UTC m=+118.736607215"
Sep 18 16:07:07.325061 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:07.325033    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:07:07.325061 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:07.325064    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:07:07.325508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:07.325489    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b20bad3b-c233-42b2-8860-2a53b864ade8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:07:07.327265 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:07.327247    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b20bad3b-c233-42b2-8860-2a53b864ade8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svv8w\" (UID: \"b20bad3b-c233-42b2-8860-2a53b864ade8\") " pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:07:07.435350 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:07.435323    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mnksp\""
Sep 18 16:07:07.442952 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:07.442933    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-svv8w"
Sep 18 16:07:07.709455 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:07.709422    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-svv8w"]
Sep 18 16:07:07.712241 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:07:07.712206    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20bad3b_c233_42b2_8860_2a53b864ade8.slice/crio-7a79095df6f1536efe309b630a74153d5ed41e20b570158d9b06a4da9f8aefc3 WatchSource:0}: Error finding container 7a79095df6f1536efe309b630a74153d5ed41e20b570158d9b06a4da9f8aefc3: Status 404 returned error can't find the container with id 7a79095df6f1536efe309b630a74153d5ed41e20b570158d9b06a4da9f8aefc3
Sep 18 16:07:07.788203 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:07.788178    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-svv8w" event={"ID":"b20bad3b-c233-42b2-8860-2a53b864ade8","Type":"ContainerStarted","Data":"e48dc6fd01f13db3bffae0e2a897d460becfddce44dfec6b88dc6b145a7489a7"}
Sep 18 16:07:07.788322 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:07.788241    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-svv8w" event={"ID":"b20bad3b-c233-42b2-8860-2a53b864ade8","Type":"ContainerStarted","Data":"7a79095df6f1536efe309b630a74153d5ed41e20b570158d9b06a4da9f8aefc3"}
Sep 18 16:07:08.792444 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:08.792413    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-svv8w" event={"ID":"b20bad3b-c233-42b2-8860-2a53b864ade8","Type":"ContainerStarted","Data":"0da34646c909ab4b3a9a620607afddf9ff99ef5f6dad1ada2a471930b27ddd5a"}
Sep 18 16:07:10.801189 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:10.801151    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-svv8w" event={"ID":"b20bad3b-c233-42b2-8860-2a53b864ade8","Type":"ContainerStarted","Data":"4f90cfae50d4931ee69548b03242984c8e43261b8f869897cf671ef8f55727bd"}
Sep 18 16:07:10.825649 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:10.825602    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-svv8w" podStartSLOduration=17.540994566 podStartE2EDuration="19.825585947s" totalImagesPullingTime="2.284591381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:06:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:10.825118567 +0000 UTC m=+125.772617342" watchObservedRunningTime="2026-09-18 16:07:10.825585947 +0000 UTC m=+125.773084720"
Sep 18 16:07:10.877492 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:10.877466    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69f947d478-zw294"]
Sep 18 16:07:17.105868 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:17.105834    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs\") pod \"network-metrics-daemon-6psxt\" (UID: \"dac4f70d-7789-4a02-8c3a-e2f28245e0ec\") " pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:07:17.108221 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:17.108197    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac4f70d-7789-4a02-8c3a-e2f28245e0ec-metrics-certs\") pod \"network-metrics-daemon-6psxt\" (UID: \"dac4f70d-7789-4a02-8c3a-e2f28245e0ec\") " pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:07:17.284997 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:17.284971    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89djg\""
Sep 18 16:07:17.293024 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:17.293007    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6psxt"
Sep 18 16:07:17.559228 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:17.559203    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6psxt"]
Sep 18 16:07:17.561873 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:07:17.561852    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac4f70d_7789_4a02_8c3a_e2f28245e0ec.slice/crio-2826d573a05df1e0e49ae0b9ac65410cca86b3cbb4fc5dd56d7bb18dd9388145 WatchSource:0}: Error finding container 2826d573a05df1e0e49ae0b9ac65410cca86b3cbb4fc5dd56d7bb18dd9388145: Status 404 returned error can't find the container with id 2826d573a05df1e0e49ae0b9ac65410cca86b3cbb4fc5dd56d7bb18dd9388145
Sep 18 16:07:17.823649 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:17.823562    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6psxt" event={"ID":"dac4f70d-7789-4a02-8c3a-e2f28245e0ec","Type":"ContainerStarted","Data":"2826d573a05df1e0e49ae0b9ac65410cca86b3cbb4fc5dd56d7bb18dd9388145"}
Sep 18 16:07:18.828343 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:18.828300    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6psxt" event={"ID":"dac4f70d-7789-4a02-8c3a-e2f28245e0ec","Type":"ContainerStarted","Data":"5972b89de2c055c1adb16ab1b47461b93b710581f829b5029d986cde199b35b1"}
Sep 18 16:07:18.828827 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:18.828408    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6psxt" event={"ID":"dac4f70d-7789-4a02-8c3a-e2f28245e0ec","Type":"ContainerStarted","Data":"f4027a9fe097c917f1316ec0ec67d0d35b458ebe460256797bae56df0dd23a5e"}
Sep 18 16:07:18.844958 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:18.844913    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6psxt" podStartSLOduration=131.039399005 podStartE2EDuration="2m11.844898303s" totalImagesPullingTime="805.499298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:18.843282614 +0000 UTC m=+133.790781404" watchObservedRunningTime="2026-09-18 16:07:18.844898303 +0000 UTC m=+133.792397108"
Sep 18 16:07:19.685557 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.685512    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"]
Sep 18 16:07:19.689467 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.689452    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"
Sep 18 16:07:19.691907 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.691885    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 18 16:07:19.692037 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.691884    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 18 16:07:19.692160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.692147    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 16:07:19.692701 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.692684    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 16:07:19.692812 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.692785    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 16:07:19.692882 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.692813    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mdlxq\""
Sep 18 16:07:19.700919 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.700900    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"]
Sep 18 16:07:19.714028 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.714011    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8m9fc"]
Sep 18 16:07:19.717800 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.717787    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.720105 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720084    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kgw4g\""
Sep 18 16:07:19.720195 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720095    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 16:07:19.720305 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720287    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18e4b877-0577-49d1-aa0c-0727ecfe14f0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-cpfp4\" (UID: \"18e4b877-0577-49d1-aa0c-0727ecfe14f0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"
Sep 18 16:07:19.720367 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720315    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e9cf752-4bea-4cc1-8b09-adb51def1591-metrics-client-ca\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.720367 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720335    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-accelerators-collector-config\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.720367 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720346    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 16:07:19.720511 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720370    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s4gs\" (UniqueName: \"kubernetes.io/projected/18e4b877-0577-49d1-aa0c-0727ecfe14f0-kube-api-access-8s4gs\") pod \"openshift-state-metrics-86c5bc79d6-cpfp4\" (UID: \"18e4b877-0577-49d1-aa0c-0727ecfe14f0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"
Sep 18 16:07:19.720511 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720389    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e9cf752-4bea-4cc1-8b09-adb51def1591-sys\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.720511 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720446    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-tls\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.720511 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720467    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e9cf752-4bea-4cc1-8b09-adb51def1591-root\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.720511 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720493    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-textfile\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.720511 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720511    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18e4b877-0577-49d1-aa0c-0727ecfe14f0-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-cpfp4\" (UID: \"18e4b877-0577-49d1-aa0c-0727ecfe14f0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"
Sep 18 16:07:19.720830 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720513    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 16:07:19.720830 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720571    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.720830 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720596    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-wtmp\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.720830 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720611    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/18e4b877-0577-49d1-aa0c-0727ecfe14f0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-cpfp4\" (UID: \"18e4b877-0577-49d1-aa0c-0727ecfe14f0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"
Sep 18 16:07:19.720830 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.720663    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzm8d\" (UniqueName: \"kubernetes.io/projected/9e9cf752-4bea-4cc1-8b09-adb51def1591-kube-api-access-vzm8d\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.821570 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.821544    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-wtmp\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.821667 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.821574    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/18e4b877-0577-49d1-aa0c-0727ecfe14f0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-cpfp4\" (UID: \"18e4b877-0577-49d1-aa0c-0727ecfe14f0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"
Sep 18 16:07:19.821667 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.821594    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzm8d\" (UniqueName: \"kubernetes.io/projected/9e9cf752-4bea-4cc1-8b09-adb51def1591-kube-api-access-vzm8d\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.821667 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.821650    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18e4b877-0577-49d1-aa0c-0727ecfe14f0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-cpfp4\" (UID: \"18e4b877-0577-49d1-aa0c-0727ecfe14f0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"
Sep 18 16:07:19.821824 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.821680    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e9cf752-4bea-4cc1-8b09-adb51def1591-metrics-client-ca\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.821824 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:07:19.821685    2554 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 18 16:07:19.821824 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.821711    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-accelerators-collector-config\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.821824 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.821727    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-wtmp\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.821824 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:07:19.821753    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e4b877-0577-49d1-aa0c-0727ecfe14f0-openshift-state-metrics-tls podName:18e4b877-0577-49d1-aa0c-0727ecfe14f0 nodeName:}" failed. No retries permitted until 2026-09-18 16:07:20.321733881 +0000 UTC m=+135.269232633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/18e4b877-0577-49d1-aa0c-0727ecfe14f0-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-cpfp4" (UID: "18e4b877-0577-49d1-aa0c-0727ecfe14f0") : secret "openshift-state-metrics-tls" not found
Sep 18 16:07:19.821824 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.821790    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s4gs\" (UniqueName: \"kubernetes.io/projected/18e4b877-0577-49d1-aa0c-0727ecfe14f0-kube-api-access-8s4gs\") pod \"openshift-state-metrics-86c5bc79d6-cpfp4\" (UID: \"18e4b877-0577-49d1-aa0c-0727ecfe14f0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"
Sep 18 16:07:19.821824 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.821817    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e9cf752-4bea-4cc1-8b09-adb51def1591-sys\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.822160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.821846    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-tls\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.822160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.821869    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e9cf752-4bea-4cc1-8b09-adb51def1591-root\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.822160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.821901    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-textfile\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.822160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.821909    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e9cf752-4bea-4cc1-8b09-adb51def1591-sys\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.822160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.821940    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18e4b877-0577-49d1-aa0c-0727ecfe14f0-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-cpfp4\" (UID: \"18e4b877-0577-49d1-aa0c-0727ecfe14f0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"
Sep 18 16:07:19.822160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.821945    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e9cf752-4bea-4cc1-8b09-adb51def1591-root\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.822160 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:07:19.821948    2554 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 16:07:19.822160 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:07:19.822012    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-tls podName:9e9cf752-4bea-4cc1-8b09-adb51def1591 nodeName:}" failed. No retries permitted until 2026-09-18 16:07:20.321996417 +0000 UTC m=+135.269495183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-tls") pod "node-exporter-8m9fc" (UID: "9e9cf752-4bea-4cc1-8b09-adb51def1591") : secret "node-exporter-tls" not found
Sep 18 16:07:19.822160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.821967    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.822514 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.822236    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-textfile\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.822514 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.822370    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e9cf752-4bea-4cc1-8b09-adb51def1591-metrics-client-ca\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.822514 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.822414    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-accelerators-collector-config\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.822862 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.822841    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18e4b877-0577-49d1-aa0c-0727ecfe14f0-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-cpfp4\" (UID: \"18e4b877-0577-49d1-aa0c-0727ecfe14f0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"
Sep 18 16:07:19.824360 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.824343    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18e4b877-0577-49d1-aa0c-0727ecfe14f0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-cpfp4\" (UID: \"18e4b877-0577-49d1-aa0c-0727ecfe14f0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"
Sep 18 16:07:19.824360 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.824354    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:19.832083 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.832060    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s4gs\" (UniqueName: \"kubernetes.io/projected/18e4b877-0577-49d1-aa0c-0727ecfe14f0-kube-api-access-8s4gs\") pod \"openshift-state-metrics-86c5bc79d6-cpfp4\" (UID: \"18e4b877-0577-49d1-aa0c-0727ecfe14f0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"
Sep 18 16:07:19.832373 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:19.832334    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzm8d\" (UniqueName: \"kubernetes.io/projected/9e9cf752-4bea-4cc1-8b09-adb51def1591-kube-api-access-vzm8d\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:20.325741 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:20.325709    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/18e4b877-0577-49d1-aa0c-0727ecfe14f0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-cpfp4\" (UID: \"18e4b877-0577-49d1-aa0c-0727ecfe14f0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"
Sep 18 16:07:20.325904 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:20.325766    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-tls\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:20.328188 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:20.328160    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e9cf752-4bea-4cc1-8b09-adb51def1591-node-exporter-tls\") pod \"node-exporter-8m9fc\" (UID: \"9e9cf752-4bea-4cc1-8b09-adb51def1591\") " pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:20.328365 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:20.328344    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/18e4b877-0577-49d1-aa0c-0727ecfe14f0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-cpfp4\" (UID: \"18e4b877-0577-49d1-aa0c-0727ecfe14f0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"
Sep 18 16:07:20.598133 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:20.598041    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"
Sep 18 16:07:20.626989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:20.626956    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8m9fc"
Sep 18 16:07:20.638024 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:07:20.637998    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9cf752_4bea_4cc1_8b09_adb51def1591.slice/crio-7eb9608e91ce8f737f2a65dbfd31a7bc18c1aef5291cd87ec1ca3997cd505dc1 WatchSource:0}: Error finding container 7eb9608e91ce8f737f2a65dbfd31a7bc18c1aef5291cd87ec1ca3997cd505dc1: Status 404 returned error can't find the container with id 7eb9608e91ce8f737f2a65dbfd31a7bc18c1aef5291cd87ec1ca3997cd505dc1
Sep 18 16:07:20.838354 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:20.838324    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8m9fc" event={"ID":"9e9cf752-4bea-4cc1-8b09-adb51def1591","Type":"ContainerStarted","Data":"7eb9608e91ce8f737f2a65dbfd31a7bc18c1aef5291cd87ec1ca3997cd505dc1"}
Sep 18 16:07:20.874473 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:20.874406    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4"]
Sep 18 16:07:20.877138 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:07:20.877115    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e4b877_0577_49d1_aa0c_0727ecfe14f0.slice/crio-1c6cabcae8b9a421e86d95273e25f5afbf4006887cd64c67c0e23c54a1a9023f WatchSource:0}: Error finding container 1c6cabcae8b9a421e86d95273e25f5afbf4006887cd64c67c0e23c54a1a9023f: Status 404 returned error can't find the container with id 1c6cabcae8b9a421e86d95273e25f5afbf4006887cd64c67c0e23c54a1a9023f
Sep 18 16:07:20.885042 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:20.885015    2554 patch_prober.go:28] interesting pod/image-registry-69f947d478-zw294 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:07:20.885134 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:20.885058    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69f947d478-zw294" podUID="c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:07:21.705426 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.705382    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"]
Sep 18 16:07:21.709450 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.709428    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.712178 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.711970    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 16:07:21.712178 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.711997    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6nsd9\""
Sep 18 16:07:21.712178 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.712033    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 16:07:21.712178 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.712096    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 16:07:21.712450 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.712276    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 16:07:21.712703 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.712683    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e8mjiuub8k25v\""
Sep 18 16:07:21.712811 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.712689    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 16:07:21.720405 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.720386    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"]
Sep 18 16:07:21.736126 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.736104    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-grpc-tls\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.736221 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.736138    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.736221 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.736167    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.736319 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.736256    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-thanos-querier-tls\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.736319 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.736299    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.736411 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.736381    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48d24\" (UniqueName: \"kubernetes.io/projected/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-kube-api-access-48d24\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.736471 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.736449    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.736541 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.736482    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-metrics-client-ca\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.837477 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.837436    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.837655 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.837483    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-metrics-client-ca\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.837655 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.837590    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-grpc-tls\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.837655 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.837624    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.837833 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.837655    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.837833 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.837694    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-thanos-querier-tls\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.837833 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.837729    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.837833 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.837779    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48d24\" (UniqueName: \"kubernetes.io/projected/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-kube-api-access-48d24\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.838330 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.838298    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-metrics-client-ca\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.840777 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.840721    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.841247 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.840796    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.841247 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.840893    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-grpc-tls\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.841247 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.840898    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.841247 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.840969    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.841679 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.841636    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-secret-thanos-querier-tls\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:21.847571 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.847546    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4" event={"ID":"18e4b877-0577-49d1-aa0c-0727ecfe14f0","Type":"ContainerStarted","Data":"4fece11683157c88a627e1b77bdf2a96785f1c2927de6e58ab58fee9a3299c89"}
Sep 18 16:07:21.847667 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.847600    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4" event={"ID":"18e4b877-0577-49d1-aa0c-0727ecfe14f0","Type":"ContainerStarted","Data":"a95f3f4d7c0dbe4bc1f0ae480e9c7f9453ec43a70cd44e188cdd01d6deed66d0"}
Sep 18 16:07:21.847667 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.847623    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4" event={"ID":"18e4b877-0577-49d1-aa0c-0727ecfe14f0","Type":"ContainerStarted","Data":"1c6cabcae8b9a421e86d95273e25f5afbf4006887cd64c67c0e23c54a1a9023f"}
Sep 18 16:07:21.848744 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:21.848726    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48d24\" (UniqueName: \"kubernetes.io/projected/938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9-kube-api-access-48d24\") pod \"thanos-querier-5dbfcdd95b-vn26r\" (UID: \"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9\") " pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:22.020831 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:22.020803    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:22.297939 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:22.297869    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"]
Sep 18 16:07:22.449886 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:07:22.449847    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938c4ab9_6f14_4a8e_9bc8_09c8e8c177d9.slice/crio-3dc4336f5e99e9bafef2862c0ca4fdd1dea696b2a8b14cf23d643bf2dbdf03b5 WatchSource:0}: Error finding container 3dc4336f5e99e9bafef2862c0ca4fdd1dea696b2a8b14cf23d643bf2dbdf03b5: Status 404 returned error can't find the container with id 3dc4336f5e99e9bafef2862c0ca4fdd1dea696b2a8b14cf23d643bf2dbdf03b5
Sep 18 16:07:22.851727 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:22.851692    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r" event={"ID":"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9","Type":"ContainerStarted","Data":"3dc4336f5e99e9bafef2862c0ca4fdd1dea696b2a8b14cf23d643bf2dbdf03b5"}
Sep 18 16:07:22.853141 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:22.853119    2554 generic.go:356] "Generic (PLEG): container finished" podID="9e9cf752-4bea-4cc1-8b09-adb51def1591" containerID="d3406d7616b930b8bb066e000d1769ea3fb855fa6a8179e86c3dd10fc18bd0ec" exitCode=0
Sep 18 16:07:22.853247 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:22.853147    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8m9fc" event={"ID":"9e9cf752-4bea-4cc1-8b09-adb51def1591","Type":"ContainerDied","Data":"d3406d7616b930b8bb066e000d1769ea3fb855fa6a8179e86c3dd10fc18bd0ec"}
Sep 18 16:07:22.855307 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:22.855290    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4" event={"ID":"18e4b877-0577-49d1-aa0c-0727ecfe14f0","Type":"ContainerStarted","Data":"e0553202f7e0f2da65fcce01c75fae776dc22c0067eb793a917ca60b15bbda02"}
Sep 18 16:07:22.885677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:22.885633    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-cpfp4" podStartSLOduration=2.385041103 podStartE2EDuration="3.88561755s" totalImagesPullingTime="1.500576447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:22.88464958 +0000 UTC m=+137.832148364" watchObservedRunningTime="2026-09-18 16:07:22.88561755 +0000 UTC m=+137.833116323"
Sep 18 16:07:23.860300 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:23.860267    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8m9fc" event={"ID":"9e9cf752-4bea-4cc1-8b09-adb51def1591","Type":"ContainerStarted","Data":"96d02d0cbf6848cd7667835302f2215d1a5cd7fac9f5b15c577f03978aa4ffa7"}
Sep 18 16:07:23.860698 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:23.860324    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8m9fc" event={"ID":"9e9cf752-4bea-4cc1-8b09-adb51def1591","Type":"ContainerStarted","Data":"e277955a3322425e78995ae38fc97d93e075e5aa59201e6648f4f30d9bdbaa00"}
Sep 18 16:07:23.884645 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:23.881563    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8m9fc" podStartSLOduration=3.575957655 podStartE2EDuration="4.881541909s" totalImagesPullingTime="1.239343049s" totalInitContainerRuntime="66.241205ms" isStatefulPod=true podCreationTimestamp="2026-09-18 16:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:23.879410365 +0000 UTC m=+138.826909136" watchObservedRunningTime="2026-09-18 16:07:23.881541909 +0000 UTC m=+138.829040676"
Sep 18 16:07:24.451403 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.451369    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-pqkvc"]
Sep 18 16:07:24.454921 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.454891    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-pqkvc"
Sep 18 16:07:24.457160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.457138    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pvs9f\""
Sep 18 16:07:24.457292 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.457178    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 18 16:07:24.463719 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.463356    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-pqkvc"]
Sep 18 16:07:24.558711 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.558681    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0fa632f-e2c2-445e-a4eb-1f37cb3047b4-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-pqkvc\" (UID: \"a0fa632f-e2c2-445e-a4eb-1f37cb3047b4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-pqkvc"
Sep 18 16:07:24.558898 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.558729    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8gxn\" (UniqueName: \"kubernetes.io/projected/a0fa632f-e2c2-445e-a4eb-1f37cb3047b4-kube-api-access-r8gxn\") pod \"monitoring-plugin-5984459ff8-pqkvc\" (UID: \"a0fa632f-e2c2-445e-a4eb-1f37cb3047b4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-pqkvc"
Sep 18 16:07:24.659431 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.659396    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0fa632f-e2c2-445e-a4eb-1f37cb3047b4-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-pqkvc\" (UID: \"a0fa632f-e2c2-445e-a4eb-1f37cb3047b4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-pqkvc"
Sep 18 16:07:24.659609 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.659442    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8gxn\" (UniqueName: \"kubernetes.io/projected/a0fa632f-e2c2-445e-a4eb-1f37cb3047b4-kube-api-access-r8gxn\") pod \"monitoring-plugin-5984459ff8-pqkvc\" (UID: \"a0fa632f-e2c2-445e-a4eb-1f37cb3047b4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-pqkvc"
Sep 18 16:07:24.662338 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.662310    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0fa632f-e2c2-445e-a4eb-1f37cb3047b4-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-pqkvc\" (UID: \"a0fa632f-e2c2-445e-a4eb-1f37cb3047b4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-pqkvc"
Sep 18 16:07:24.667346 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.667319    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8gxn\" (UniqueName: \"kubernetes.io/projected/a0fa632f-e2c2-445e-a4eb-1f37cb3047b4-kube-api-access-r8gxn\") pod \"monitoring-plugin-5984459ff8-pqkvc\" (UID: \"a0fa632f-e2c2-445e-a4eb-1f37cb3047b4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-pqkvc"
Sep 18 16:07:24.765356 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.765277    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-pqkvc"
Sep 18 16:07:24.897619 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.897325    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-59478d44bc-x578z"]
Sep 18 16:07:24.902000 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.901980    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:24.904581 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.904558    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 16:07:24.905023 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.905004    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8gwjv\""
Sep 18 16:07:24.905125 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.905048    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 16:07:24.905384 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.905369    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 16:07:24.905447 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.905393    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 16:07:24.905782 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.905752    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 16:07:24.911698 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.911678    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 16:07:24.918508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.918487    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-59478d44bc-x578z"]
Sep 18 16:07:24.961692 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.961661    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2jvf\" (UniqueName: \"kubernetes.io/projected/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-kube-api-access-t2jvf\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:24.961793 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.961744    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-serving-certs-ca-bundle\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:24.961793 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.961769    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:24.961903 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.961822    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-telemeter-client-tls\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:24.961903 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.961870    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-secret-telemeter-client\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:24.962006 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.961903    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-metrics-client-ca\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:24.962006 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.962001    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-federate-client-tls\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:24.962111 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:24.962032    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.058439 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.058378    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-pqkvc"]
Sep 18 16:07:25.060954 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:07:25.060932    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0fa632f_e2c2_445e_a4eb_1f37cb3047b4.slice/crio-4043246b51b3157c59ad2c4d4fa4db0114b3f2b7f95e5c31175003f86459118b WatchSource:0}: Error finding container 4043246b51b3157c59ad2c4d4fa4db0114b3f2b7f95e5c31175003f86459118b: Status 404 returned error can't find the container with id 4043246b51b3157c59ad2c4d4fa4db0114b3f2b7f95e5c31175003f86459118b
Sep 18 16:07:25.062563 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.062496    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-serving-certs-ca-bundle\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.062621 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.062558    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.062621 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.062602    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-telemeter-client-tls\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.062786 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.062633    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-secret-telemeter-client\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.062786 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.062669    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-metrics-client-ca\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.062786 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.062731    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-federate-client-tls\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.062786 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.062760    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.062978 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.062796    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2jvf\" (UniqueName: \"kubernetes.io/projected/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-kube-api-access-t2jvf\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.063402 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.063377    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-serving-certs-ca-bundle\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.063515 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.063390    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-metrics-client-ca\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.063515 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.063403    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.065582 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.065558    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-federate-client-tls\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.065673 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.065630    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.065870 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.065851    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-secret-telemeter-client\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.065933 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.065851    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-telemeter-client-tls\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.071357 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.071332    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2jvf\" (UniqueName: \"kubernetes.io/projected/924ab2c9-fbfc-4492-b97b-13d6a2bb1b16-kube-api-access-t2jvf\") pod \"telemeter-client-59478d44bc-x578z\" (UID: \"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16\") " pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.225970 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.225948    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-59478d44bc-x578z"
Sep 18 16:07:25.499401 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.499371    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-59478d44bc-x578z"]
Sep 18 16:07:25.501980 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:07:25.501955    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924ab2c9_fbfc_4492_b97b_13d6a2bb1b16.slice/crio-eb584fb0fb534f5aa1aca3af88316db1dfcf7e64d5011faa3454364e2809b921 WatchSource:0}: Error finding container eb584fb0fb534f5aa1aca3af88316db1dfcf7e64d5011faa3454364e2809b921: Status 404 returned error can't find the container with id eb584fb0fb534f5aa1aca3af88316db1dfcf7e64d5011faa3454364e2809b921
Sep 18 16:07:25.870947 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.870912    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r" event={"ID":"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9","Type":"ContainerStarted","Data":"afe34c3920837d62acbba22ed0de6e263cf10ca1d799ac31bee7dda427398f2d"}
Sep 18 16:07:25.871061 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.870998    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r" event={"ID":"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9","Type":"ContainerStarted","Data":"5bf04f6871eefcee445806cffc3e08857cb53877bfca35ddc763d9003fd92660"}
Sep 18 16:07:25.871061 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.871041    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r" event={"ID":"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9","Type":"ContainerStarted","Data":"aa9dc45be7cd4bfba68c33babc2d45735ff5a797f5648265fc899fdffda48847"}
Sep 18 16:07:25.872586 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.872557    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59478d44bc-x578z" event={"ID":"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16","Type":"ContainerStarted","Data":"eb584fb0fb534f5aa1aca3af88316db1dfcf7e64d5011faa3454364e2809b921"}
Sep 18 16:07:25.873917 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.873893    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-pqkvc" event={"ID":"a0fa632f-e2c2-445e-a4eb-1f37cb3047b4","Type":"ContainerStarted","Data":"4043246b51b3157c59ad2c4d4fa4db0114b3f2b7f95e5c31175003f86459118b"}
Sep 18 16:07:25.921252 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.917625    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:07:25.923493 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.923470    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.926030 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.926008    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 16:07:25.926254 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.926237    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 16:07:25.927051 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.927028    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 16:07:25.927366 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.927352    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 16:07:25.927890 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.927869    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 16:07:25.928127 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.928093    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 16:07:25.928540 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.928502    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rnhrt\""
Sep 18 16:07:25.928626 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.928549    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 16:07:25.928626 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.928581    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cduku6kld38gr\""
Sep 18 16:07:25.928726 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.928680    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 16:07:25.928726 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.928586    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 16:07:25.928982 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.928964    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 16:07:25.929170 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.929153    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 16:07:25.929335 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.929318    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 16:07:25.929592 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.929577    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 16:07:25.933158 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.933057    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 16:07:25.935718 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.935698    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:07:25.936131 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.936111    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 16:07:25.971620 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971337    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.971620 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971376    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.971620 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971403    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.971620 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971442    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.971620 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971461    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggpd6\" (UniqueName: \"kubernetes.io/projected/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-kube-api-access-ggpd6\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.971620 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971538    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-config\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.971620 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971563    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.971620 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971581    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.971620 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971598    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.971620 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971622    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.972165 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971669    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.972165 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971751    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.972165 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971789    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.972165 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971819    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.972165 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971852    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.972165 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971884    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.972165 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971913    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.972165 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971937    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.972165 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971964    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:25.972165 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:25.971996    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073080 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073004    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073080 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073051    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073280 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073096    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073280 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073121    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggpd6\" (UniqueName: \"kubernetes.io/projected/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-kube-api-access-ggpd6\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073280 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073163    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-config\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073280 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073195    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073280 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073222    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073280 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073245    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073280 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073271    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073643 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073300    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073643 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073334    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073643 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073374    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073643 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073409    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073643 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073441    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073643 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073471    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073643 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073570    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073643 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073598    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.073643 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073628    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.074074 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073662    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.074074 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073698    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.074074 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.073783    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.074344 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.074320    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.074915 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.074886    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.075009 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.074928    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.076193 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.075596    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.076193 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.075883    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.077661 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.077635    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.077748 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.077684    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.078235 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.078191    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.079023 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.078980    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.082289 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.081910    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.082289 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.081976    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.082289 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.081982    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.082289 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.082090    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.082289 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.082210    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.082289 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.082255    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.082624 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.082364    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.082624 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.082445    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-config\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.083861 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.083688    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.085999 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.085980    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggpd6\" (UniqueName: \"kubernetes.io/projected/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-kube-api-access-ggpd6\") pod \"prometheus-k8s-0\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.248070 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.247819    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:26.740983 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.740897    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:07:26.880581 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.880543    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-pqkvc" event={"ID":"a0fa632f-e2c2-445e-a4eb-1f37cb3047b4","Type":"ContainerStarted","Data":"78c5057fe54eb4c1fda7530af723283dedbba1ec22b263bf0cc6a1c1ef2c3f02"}
Sep 18 16:07:26.880766 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.880608    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-pqkvc"
Sep 18 16:07:26.884456 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.884415    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r" event={"ID":"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9","Type":"ContainerStarted","Data":"d94c78627cfa6ef231c023bd3008b6a0b335c24a74ba7b0065a4068172a0a816"}
Sep 18 16:07:26.884632 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.884492    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r" event={"ID":"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9","Type":"ContainerStarted","Data":"959fb97ed240db334eafb8ba3878b13fa957dbdaa2b9b9b6b4502ce4652c0da2"}
Sep 18 16:07:26.884632 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.884546    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r" event={"ID":"938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9","Type":"ContainerStarted","Data":"f191ed9a9d91270809edb657816b1d6a15995c184948f1a9f6cc14e0e9129992"}
Sep 18 16:07:26.884809 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.884787    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:26.886050 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.886030    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-pqkvc"
Sep 18 16:07:26.895638 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.895576    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-pqkvc" podStartSLOduration=1.462775838 podStartE2EDuration="2.895563216s" totalImagesPullingTime="1.432787378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:26.894586452 +0000 UTC m=+141.842085226" watchObservedRunningTime="2026-09-18 16:07:26.895563216 +0000 UTC m=+141.843061991"
Sep 18 16:07:26.915244 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:26.915206    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r" podStartSLOduration=2.7600616430000002 podStartE2EDuration="5.915192858s" totalImagesPullingTime="3.155131215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:26.913458862 +0000 UTC m=+141.860957638" watchObservedRunningTime="2026-09-18 16:07:26.915192858 +0000 UTC m=+141.862691632"
Sep 18 16:07:27.071550 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:07:27.071458    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9ffc5c_f0cd_491f_919d_6e7182b61d86.slice/crio-76aa8be3283318fff3c263770e30c6cede7836b6cd23ae0228033bfb053fc295 WatchSource:0}: Error finding container 76aa8be3283318fff3c263770e30c6cede7836b6cd23ae0228033bfb053fc295: Status 404 returned error can't find the container with id 76aa8be3283318fff3c263770e30c6cede7836b6cd23ae0228033bfb053fc295
Sep 18 16:07:27.889959 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:27.889915    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59478d44bc-x578z" event={"ID":"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16","Type":"ContainerStarted","Data":"518d8601653da3689d979a05dbf47798524e51c349e1679c561939c687590f29"}
Sep 18 16:07:27.891491 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:27.891461    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc9ffc5c-f0cd-491f-919d-6e7182b61d86","Type":"ContainerStarted","Data":"76aa8be3283318fff3c263770e30c6cede7836b6cd23ae0228033bfb053fc295"}
Sep 18 16:07:28.897608 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:28.897569    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59478d44bc-x578z" event={"ID":"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16","Type":"ContainerStarted","Data":"34e4cefe7ff2e77a6c8b55c155a9dd67562ead8ee099fc75f49ed6c82bba29da"}
Sep 18 16:07:28.898032 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:28.897635    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59478d44bc-x578z" event={"ID":"924ab2c9-fbfc-4492-b97b-13d6a2bb1b16","Type":"ContainerStarted","Data":"cf13a7311960af3c154dfd1651bca913ad04bcd222e82fb147fda17f981884a1"}
Sep 18 16:07:28.899089 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:28.899061    2554 generic.go:356] "Generic (PLEG): container finished" podID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerID="2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e" exitCode=0
Sep 18 16:07:28.899186 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:28.899087    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc9ffc5c-f0cd-491f-919d-6e7182b61d86","Type":"ContainerDied","Data":"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e"}
Sep 18 16:07:28.919283 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:28.919245    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-59478d44bc-x578z" podStartSLOduration=2.394161292 podStartE2EDuration="4.919232795s" totalImagesPullingTime="2.525071503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:28.918915819 +0000 UTC m=+143.866414595" watchObservedRunningTime="2026-09-18 16:07:28.919232795 +0000 UTC m=+143.866731569"
Sep 18 16:07:30.883310 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:30.883271    2554 patch_prober.go:28] interesting pod/image-registry-69f947d478-zw294 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:07:30.883728 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:30.883330    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69f947d478-zw294" podUID="c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:07:32.898106 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:32.898079    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5dbfcdd95b-vn26r"
Sep 18 16:07:32.918745 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:32.918715    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc9ffc5c-f0cd-491f-919d-6e7182b61d86","Type":"ContainerStarted","Data":"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555"}
Sep 18 16:07:32.918878 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:32.918774    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc9ffc5c-f0cd-491f-919d-6e7182b61d86","Type":"ContainerStarted","Data":"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd"}
Sep 18 16:07:32.918878 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:32.918799    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc9ffc5c-f0cd-491f-919d-6e7182b61d86","Type":"ContainerStarted","Data":"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10"}
Sep 18 16:07:32.918878 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:32.918827    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc9ffc5c-f0cd-491f-919d-6e7182b61d86","Type":"ContainerStarted","Data":"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b"}
Sep 18 16:07:32.918878 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:32.918855    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc9ffc5c-f0cd-491f-919d-6e7182b61d86","Type":"ContainerStarted","Data":"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e"}
Sep 18 16:07:32.919036 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:32.918884    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc9ffc5c-f0cd-491f-919d-6e7182b61d86","Type":"ContainerStarted","Data":"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d"}
Sep 18 16:07:32.946851 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:32.946783    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.787464834 podStartE2EDuration="7.946761639s" totalImagesPullingTime="4.135209351s" totalInitContainerRuntime="24.087454ms" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:32.945796286 +0000 UTC m=+147.893295061" watchObservedRunningTime="2026-09-18 16:07:32.946761639 +0000 UTC m=+147.894260414"
Sep 18 16:07:35.896584 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:35.896512    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69f947d478-zw294" podUID="c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7" containerName="registry" containerID="cri-o://7d8c2c094521202c613f43551221ba523d029f897cde76c902ee24b8b25691bd" gracePeriod=30
Sep 18 16:07:36.136376 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.136351    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:07:36.248459 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.248423    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:07:36.270370 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.270343    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-installation-pull-secrets\") pod \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") "
Sep 18 16:07:36.270477 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.270385    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-ca-trust-extracted\") pod \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") "
Sep 18 16:07:36.270477 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.270409    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-trusted-ca\") pod \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") "
Sep 18 16:07:36.270477 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.270432    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-certificates\") pod \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") "
Sep 18 16:07:36.270477 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.270458    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8s9n\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-kube-api-access-n8s9n\") pod \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") "
Sep 18 16:07:36.271119 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.270483    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls\") pod \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") "
Sep 18 16:07:36.271119 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.270499    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-bound-sa-token\") pod \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") "
Sep 18 16:07:36.271119 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.270537    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-image-registry-private-configuration\") pod \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\" (UID: \"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7\") "
Sep 18 16:07:36.271119 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.270893    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7" (UID: "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:07:36.271119 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.270969    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7" (UID: "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:07:36.273110 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.273080    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-kube-api-access-n8s9n" (OuterVolumeSpecName: "kube-api-access-n8s9n") pod "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7" (UID: "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7"). InnerVolumeSpecName "kube-api-access-n8s9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:07:36.273110 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.273094    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7" (UID: "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:07:36.273370 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.273341    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7" (UID: "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:07:36.273480 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.273429    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7" (UID: "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:07:36.273559 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.273486    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7" (UID: "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:07:36.279371 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.279346    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7" (UID: "c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:07:36.372376 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.372312    2554 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-installation-pull-secrets\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:07:36.372376 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.372350    2554 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-ca-trust-extracted\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:07:36.372376 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.372372    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-trusted-ca\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:07:36.372676 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.372391    2554 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-certificates\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:07:36.372676 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.372405    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8s9n\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-kube-api-access-n8s9n\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:07:36.372676 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.372423    2554 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-registry-tls\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:07:36.372676 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.372437    2554 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-bound-sa-token\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:07:36.372676 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.372451    2554 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7-image-registry-private-configuration\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:07:36.935783 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.935752    2554 generic.go:356] "Generic (PLEG): container finished" podID="c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7" containerID="7d8c2c094521202c613f43551221ba523d029f897cde76c902ee24b8b25691bd" exitCode=0
Sep 18 16:07:36.936268 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.935798    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f947d478-zw294"
Sep 18 16:07:36.936268 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.935812    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f947d478-zw294" event={"ID":"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7","Type":"ContainerDied","Data":"7d8c2c094521202c613f43551221ba523d029f897cde76c902ee24b8b25691bd"}
Sep 18 16:07:36.936268 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.935846    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f947d478-zw294" event={"ID":"c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7","Type":"ContainerDied","Data":"af316fcc718efccf43a3eebcc0706b208fc750828058f259c4122b0f54c5b5b8"}
Sep 18 16:07:36.936268 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.935876    2554 scope.go:117] "RemoveContainer" containerID="7d8c2c094521202c613f43551221ba523d029f897cde76c902ee24b8b25691bd"
Sep 18 16:07:36.944962 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.944945    2554 scope.go:117] "RemoveContainer" containerID="7d8c2c094521202c613f43551221ba523d029f897cde76c902ee24b8b25691bd"
Sep 18 16:07:36.945257 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:07:36.945236    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d8c2c094521202c613f43551221ba523d029f897cde76c902ee24b8b25691bd\": container with ID starting with 7d8c2c094521202c613f43551221ba523d029f897cde76c902ee24b8b25691bd not found: ID does not exist" containerID="7d8c2c094521202c613f43551221ba523d029f897cde76c902ee24b8b25691bd"
Sep 18 16:07:36.945334 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.945267    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8c2c094521202c613f43551221ba523d029f897cde76c902ee24b8b25691bd"} err="failed to get container status \"7d8c2c094521202c613f43551221ba523d029f897cde76c902ee24b8b25691bd\": rpc error: code = NotFound desc = could not find container \"7d8c2c094521202c613f43551221ba523d029f897cde76c902ee24b8b25691bd\": container with ID starting with 7d8c2c094521202c613f43551221ba523d029f897cde76c902ee24b8b25691bd not found: ID does not exist"
Sep 18 16:07:36.957174 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.957140    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69f947d478-zw294"]
Sep 18 16:07:36.961599 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:36.961578    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69f947d478-zw294"]
Sep 18 16:07:37.276253 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:37.276172    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7" path="/var/lib/kubelet/pods/c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7/volumes"
Sep 18 16:07:44.103856 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:07:44.103798    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xxw4k" podUID="cca8c0d4-a36f-4622-8f97-e18b71343f72"
Sep 18 16:07:44.966892 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:44.966865    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:07:48.972755 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:48.972667    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:07:48.975057 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:48.975033    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cca8c0d4-a36f-4622-8f97-e18b71343f72-metrics-tls\") pod \"dns-default-xxw4k\" (UID: \"cca8c0d4-a36f-4622-8f97-e18b71343f72\") " pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:07:49.170214 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:49.170187    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bgqnj\""
Sep 18 16:07:49.177725 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:49.177705    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:07:49.450789 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:49.450757    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xxw4k"]
Sep 18 16:07:49.453915 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:07:49.453887    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca8c0d4_a36f_4622_8f97_e18b71343f72.slice/crio-2819037213d64c0fdb2e308336942909a8ccc29effdbb60a9e0f08ff52a9910e WatchSource:0}: Error finding container 2819037213d64c0fdb2e308336942909a8ccc29effdbb60a9e0f08ff52a9910e: Status 404 returned error can't find the container with id 2819037213d64c0fdb2e308336942909a8ccc29effdbb60a9e0f08ff52a9910e
Sep 18 16:07:49.988718 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:49.988680    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxw4k" event={"ID":"cca8c0d4-a36f-4622-8f97-e18b71343f72","Type":"ContainerStarted","Data":"2819037213d64c0fdb2e308336942909a8ccc29effdbb60a9e0f08ff52a9910e"}
Sep 18 16:07:51.997824 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:51.997782    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxw4k" event={"ID":"cca8c0d4-a36f-4622-8f97-e18b71343f72","Type":"ContainerStarted","Data":"2925866f88e2d44107e6526b183acb17679857280422fabe27ee0d15a0f2fa6f"}
Sep 18 16:07:51.998258 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:51.997837    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxw4k" event={"ID":"cca8c0d4-a36f-4622-8f97-e18b71343f72","Type":"ContainerStarted","Data":"360695f1b6a8726260f8d581d5d3a79cf2703e461a5ca0e2f854173e6e7aa4c0"}
Sep 18 16:07:51.998258 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:51.997890    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:07:52.016348 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:07:52.016297    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xxw4k" podStartSLOduration=129.480589438 podStartE2EDuration="2m11.01628264s" totalImagesPullingTime="1.535693202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:52.014372828 +0000 UTC m=+166.961871602" watchObservedRunningTime="2026-09-18 16:07:52.01628264 +0000 UTC m=+166.963781413"
Sep 18 16:08:02.004012 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:02.003986    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xxw4k"
Sep 18 16:08:07.604343 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:07.604320    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xxw4k_cca8c0d4-a36f-4622-8f97-e18b71343f72/dns/0.log"
Sep 18 16:08:07.804861 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:07.804830    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xxw4k_cca8c0d4-a36f-4622-8f97-e18b71343f72/kube-rbac-proxy/0.log"
Sep 18 16:08:08.407481 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:08.407456    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9286k_8906b592-b7c8-46a7-ab14-c89b0803fd8d/dns-node-resolver/0.log"
Sep 18 16:08:09.404477 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:09.404447    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-26zfw_4d012abb-e750-4643-b3fe-8cffd9723e76/node-ca/0.log"
Sep 18 16:08:10.074054 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:10.073974    2554 generic.go:356] "Generic (PLEG): container finished" podID="8a5a8e40-2a6b-4289-be46-bf1e50af868e" containerID="7d4d7a26450a09481d5170ae2708f66657cc9d83bb0cb0c792a89d1468cd61db" exitCode=0
Sep 18 16:08:10.074054 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:10.074047    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp" event={"ID":"8a5a8e40-2a6b-4289-be46-bf1e50af868e","Type":"ContainerDied","Data":"7d4d7a26450a09481d5170ae2708f66657cc9d83bb0cb0c792a89d1468cd61db"}
Sep 18 16:08:10.074401 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:10.074387    2554 scope.go:117] "RemoveContainer" containerID="7d4d7a26450a09481d5170ae2708f66657cc9d83bb0cb0c792a89d1468cd61db"
Sep 18 16:08:10.204342 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:10.204308    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gsxcc_09f70d2c-590d-4474-ac55-e55056a054cd/serve-healthcheck-canary/0.log"
Sep 18 16:08:11.079702 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:11.079658    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp" event={"ID":"8a5a8e40-2a6b-4289-be46-bf1e50af868e","Type":"ContainerStarted","Data":"1720953806fefc41b2941d2b1433a47d5d6d88ee767d6cb388c5183333b10358"}
Sep 18 16:08:26.248242 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:26.248199    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:26.268075 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:26.268050    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:27.160374 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:27.160341    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:44.266541 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.266495    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:08:44.266954 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.266804    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="prometheus" containerID="cri-o://fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d" gracePeriod=600
Sep 18 16:08:44.266954 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.266854    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="kube-rbac-proxy-thanos" containerID="cri-o://8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555" gracePeriod=600
Sep 18 16:08:44.266954 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.266864    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="config-reloader" containerID="cri-o://57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e" gracePeriod=600
Sep 18 16:08:44.266954 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.266879    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="thanos-sidecar" containerID="cri-o://f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b" gracePeriod=600
Sep 18 16:08:44.266954 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.266894    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="kube-rbac-proxy-web" containerID="cri-o://63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10" gracePeriod=600
Sep 18 16:08:44.267200 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.266830    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="kube-rbac-proxy" containerID="cri-o://c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd" gracePeriod=600
Sep 18 16:08:44.501720 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.501691    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:44.615727 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.615682    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-metrics-client-ca\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.615727 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.615736    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-metrics-client-certs\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.615982 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.615766    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-kubelet-serving-ca-bundle\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.615982 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.615790    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-thanos-prometheus-http-client-file\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.615982 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.615839    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.615982 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.615868    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-tls-assets\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.615982 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.615894    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-1\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.615982 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.615930    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-kube-rbac-proxy\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.615982 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.615957    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-config\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.615982 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.615980    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-tls\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.616372 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.616008    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.616372 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.616045    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggpd6\" (UniqueName: \"kubernetes.io/projected/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-kube-api-access-ggpd6\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.616372 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.616070    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-grpc-tls\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.616372 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.616098    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-trusted-ca-bundle\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.616372 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.616122    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-0\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.616372 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.616137    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:08:44.616372 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.616148    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-db\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.616372 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.616177    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-config-out\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.616372 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.616201    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-2\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.616372 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.616236    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-web-config\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.616372 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.616242    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:08:44.616372 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.616265    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-serving-certs-ca-bundle\") pod \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\" (UID: \"cc9ffc5c-f0cd-491f-919d-6e7182b61d86\") "
Sep 18 16:08:44.617184 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.616494    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-metrics-client-ca\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.617184 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.616510    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.617835 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.617792    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:08:44.617992 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.617958    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:08:44.619203 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.618855    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:08:44.619203 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.618875    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:08:44.619203 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.618954    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:08:44.619203 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.618969    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:08:44.619203 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.618985    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:08:44.619512 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.619233    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:08:44.619512 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.619243    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:08:44.619664 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.619594    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:08:44.619881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.619796    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:08:44.620400 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.620373    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-config-out" (OuterVolumeSpecName: "config-out") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:08:44.620623 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.620591    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-kube-api-access-ggpd6" (OuterVolumeSpecName: "kube-api-access-ggpd6") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "kube-api-access-ggpd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:08:44.620907 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.620882    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:08:44.621043 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.621026    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-config" (OuterVolumeSpecName: "config") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:08:44.621264 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.621247    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:08:44.621809 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.621792    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:08:44.632275 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.632253    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-web-config" (OuterVolumeSpecName: "web-config") pod "cc9ffc5c-f0cd-491f-919d-6e7182b61d86" (UID: "cc9ffc5c-f0cd-491f-919d-6e7182b61d86"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:08:44.717815 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717781    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717815 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717814    2554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-tls-assets\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717824    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717834    2554 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-kube-rbac-proxy\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717844    2554 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-config\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717855    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717864    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717872    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggpd6\" (UniqueName: \"kubernetes.io/projected/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-kube-api-access-ggpd6\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717880    2554 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-grpc-tls\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717888    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717898    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717905    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-db\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717913    2554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-config-out\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717921    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717928    2554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-web-config\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717936    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717943    2554 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-secret-metrics-client-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:44.717979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:44.717952    2554 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc9ffc5c-f0cd-491f-919d-6e7182b61d86-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:08:45.212947 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.212916    2554 generic.go:356] "Generic (PLEG): container finished" podID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerID="8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555" exitCode=0
Sep 18 16:08:45.212947 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.212940    2554 generic.go:356] "Generic (PLEG): container finished" podID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerID="c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd" exitCode=0
Sep 18 16:08:45.212947 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.212946    2554 generic.go:356] "Generic (PLEG): container finished" podID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerID="63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10" exitCode=0
Sep 18 16:08:45.212947 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.212951    2554 generic.go:356] "Generic (PLEG): container finished" podID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerID="f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b" exitCode=0
Sep 18 16:08:45.212947 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.212956    2554 generic.go:356] "Generic (PLEG): container finished" podID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerID="57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e" exitCode=0
Sep 18 16:08:45.213227 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.212962    2554 generic.go:356] "Generic (PLEG): container finished" podID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerID="fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d" exitCode=0
Sep 18 16:08:45.213227 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.212989    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc9ffc5c-f0cd-491f-919d-6e7182b61d86","Type":"ContainerDied","Data":"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555"}
Sep 18 16:08:45.213227 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.213002    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.213227 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.213044    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc9ffc5c-f0cd-491f-919d-6e7182b61d86","Type":"ContainerDied","Data":"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd"}
Sep 18 16:08:45.213227 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.213071    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc9ffc5c-f0cd-491f-919d-6e7182b61d86","Type":"ContainerDied","Data":"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10"}
Sep 18 16:08:45.213227 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.213095    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc9ffc5c-f0cd-491f-919d-6e7182b61d86","Type":"ContainerDied","Data":"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b"}
Sep 18 16:08:45.213227 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.213120    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc9ffc5c-f0cd-491f-919d-6e7182b61d86","Type":"ContainerDied","Data":"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e"}
Sep 18 16:08:45.213227 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.213123    2554 scope.go:117] "RemoveContainer" containerID="8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555"
Sep 18 16:08:45.213227 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.213143    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc9ffc5c-f0cd-491f-919d-6e7182b61d86","Type":"ContainerDied","Data":"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d"}
Sep 18 16:08:45.213227 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.213167    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc9ffc5c-f0cd-491f-919d-6e7182b61d86","Type":"ContainerDied","Data":"76aa8be3283318fff3c263770e30c6cede7836b6cd23ae0228033bfb053fc295"}
Sep 18 16:08:45.221045 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.221025    2554 scope.go:117] "RemoveContainer" containerID="c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd"
Sep 18 16:08:45.227994 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.227978    2554 scope.go:117] "RemoveContainer" containerID="63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10"
Sep 18 16:08:45.234699 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.234642    2554 scope.go:117] "RemoveContainer" containerID="f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b"
Sep 18 16:08:45.239213 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.239192    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:08:45.241482 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.241469    2554 scope.go:117] "RemoveContainer" containerID="57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e"
Sep 18 16:08:45.247220 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.247193    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:08:45.249860 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.249845    2554 scope.go:117] "RemoveContainer" containerID="fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d"
Sep 18 16:08:45.256967 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.256949    2554 scope.go:117] "RemoveContainer" containerID="2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e"
Sep 18 16:08:45.263447 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.263431    2554 scope.go:117] "RemoveContainer" containerID="8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555"
Sep 18 16:08:45.263829 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:08:45.263798    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555\": container with ID starting with 8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555 not found: ID does not exist" containerID="8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555"
Sep 18 16:08:45.263901 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.263837    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555"} err="failed to get container status \"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555\": rpc error: code = NotFound desc = could not find container \"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555\": container with ID starting with 8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555 not found: ID does not exist"
Sep 18 16:08:45.263901 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.263853    2554 scope.go:117] "RemoveContainer" containerID="c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd"
Sep 18 16:08:45.264115 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:08:45.264089    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd\": container with ID starting with c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd not found: ID does not exist" containerID="c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd"
Sep 18 16:08:45.264164 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.264125    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd"} err="failed to get container status \"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd\": rpc error: code = NotFound desc = could not find container \"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd\": container with ID starting with c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd not found: ID does not exist"
Sep 18 16:08:45.264164 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.264147    2554 scope.go:117] "RemoveContainer" containerID="63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10"
Sep 18 16:08:45.264413 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:08:45.264396    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10\": container with ID starting with 63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10 not found: ID does not exist" containerID="63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10"
Sep 18 16:08:45.264452 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.264417    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10"} err="failed to get container status \"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10\": rpc error: code = NotFound desc = could not find container \"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10\": container with ID starting with 63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10 not found: ID does not exist"
Sep 18 16:08:45.264452 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.264432    2554 scope.go:117] "RemoveContainer" containerID="f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b"
Sep 18 16:08:45.264771 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:08:45.264751    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b\": container with ID starting with f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b not found: ID does not exist" containerID="f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b"
Sep 18 16:08:45.264846 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.264778    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b"} err="failed to get container status \"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b\": rpc error: code = NotFound desc = could not find container \"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b\": container with ID starting with f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b not found: ID does not exist"
Sep 18 16:08:45.264846 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.264803    2554 scope.go:117] "RemoveContainer" containerID="57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e"
Sep 18 16:08:45.265066 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:08:45.265049    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e\": container with ID starting with 57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e not found: ID does not exist" containerID="57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e"
Sep 18 16:08:45.265104 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.265073    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e"} err="failed to get container status \"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e\": rpc error: code = NotFound desc = could not find container \"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e\": container with ID starting with 57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e not found: ID does not exist"
Sep 18 16:08:45.265104 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.265088    2554 scope.go:117] "RemoveContainer" containerID="fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d"
Sep 18 16:08:45.265314 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:08:45.265293    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d\": container with ID starting with fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d not found: ID does not exist" containerID="fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d"
Sep 18 16:08:45.265359 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.265317    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d"} err="failed to get container status \"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d\": rpc error: code = NotFound desc = could not find container \"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d\": container with ID starting with fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d not found: ID does not exist"
Sep 18 16:08:45.265359 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.265331    2554 scope.go:117] "RemoveContainer" containerID="2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e"
Sep 18 16:08:45.265590 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:08:45.265567    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e\": container with ID starting with 2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e not found: ID does not exist" containerID="2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e"
Sep 18 16:08:45.265678 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.265593    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e"} err="failed to get container status \"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e\": rpc error: code = NotFound desc = could not find container \"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e\": container with ID starting with 2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e not found: ID does not exist"
Sep 18 16:08:45.265678 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.265607    2554 scope.go:117] "RemoveContainer" containerID="8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555"
Sep 18 16:08:45.265840 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.265821    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555"} err="failed to get container status \"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555\": rpc error: code = NotFound desc = could not find container \"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555\": container with ID starting with 8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555 not found: ID does not exist"
Sep 18 16:08:45.265885 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.265843    2554 scope.go:117] "RemoveContainer" containerID="c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd"
Sep 18 16:08:45.266038 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.266020    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd"} err="failed to get container status \"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd\": rpc error: code = NotFound desc = could not find container \"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd\": container with ID starting with c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd not found: ID does not exist"
Sep 18 16:08:45.266038 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.266036    2554 scope.go:117] "RemoveContainer" containerID="63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10"
Sep 18 16:08:45.266231 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.266216    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10"} err="failed to get container status \"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10\": rpc error: code = NotFound desc = could not find container \"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10\": container with ID starting with 63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10 not found: ID does not exist"
Sep 18 16:08:45.266273 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.266231    2554 scope.go:117] "RemoveContainer" containerID="f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b"
Sep 18 16:08:45.266432 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.266417    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b"} err="failed to get container status \"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b\": rpc error: code = NotFound desc = could not find container \"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b\": container with ID starting with f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b not found: ID does not exist"
Sep 18 16:08:45.266470 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.266431    2554 scope.go:117] "RemoveContainer" containerID="57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e"
Sep 18 16:08:45.266641 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.266624    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e"} err="failed to get container status \"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e\": rpc error: code = NotFound desc = could not find container \"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e\": container with ID starting with 57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e not found: ID does not exist"
Sep 18 16:08:45.266902 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.266641    2554 scope.go:117] "RemoveContainer" containerID="fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d"
Sep 18 16:08:45.266902 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.266816    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d"} err="failed to get container status \"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d\": rpc error: code = NotFound desc = could not find container \"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d\": container with ID starting with fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d not found: ID does not exist"
Sep 18 16:08:45.266902 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.266828    2554 scope.go:117] "RemoveContainer" containerID="2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e"
Sep 18 16:08:45.267072 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.267058    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e"} err="failed to get container status \"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e\": rpc error: code = NotFound desc = could not find container \"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e\": container with ID starting with 2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e not found: ID does not exist"
Sep 18 16:08:45.267072 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.267072    2554 scope.go:117] "RemoveContainer" containerID="8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555"
Sep 18 16:08:45.267249 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.267234    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555"} err="failed to get container status \"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555\": rpc error: code = NotFound desc = could not find container \"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555\": container with ID starting with 8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555 not found: ID does not exist"
Sep 18 16:08:45.267249 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.267248    2554 scope.go:117] "RemoveContainer" containerID="c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd"
Sep 18 16:08:45.267422 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.267406    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd"} err="failed to get container status \"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd\": rpc error: code = NotFound desc = could not find container \"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd\": container with ID starting with c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd not found: ID does not exist"
Sep 18 16:08:45.267465 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.267422    2554 scope.go:117] "RemoveContainer" containerID="63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10"
Sep 18 16:08:45.267630 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.267615    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10"} err="failed to get container status \"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10\": rpc error: code = NotFound desc = could not find container \"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10\": container with ID starting with 63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10 not found: ID does not exist"
Sep 18 16:08:45.267677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.267630    2554 scope.go:117] "RemoveContainer" containerID="f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b"
Sep 18 16:08:45.267820 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.267796    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b"} err="failed to get container status \"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b\": rpc error: code = NotFound desc = could not find container \"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b\": container with ID starting with f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b not found: ID does not exist"
Sep 18 16:08:45.267866 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.267822    2554 scope.go:117] "RemoveContainer" containerID="57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e"
Sep 18 16:08:45.268037 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.268022    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e"} err="failed to get container status \"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e\": rpc error: code = NotFound desc = could not find container \"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e\": container with ID starting with 57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e not found: ID does not exist"
Sep 18 16:08:45.268037 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.268036    2554 scope.go:117] "RemoveContainer" containerID="fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d"
Sep 18 16:08:45.268255 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.268239    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d"} err="failed to get container status \"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d\": rpc error: code = NotFound desc = could not find container \"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d\": container with ID starting with fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d not found: ID does not exist"
Sep 18 16:08:45.268306 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.268256    2554 scope.go:117] "RemoveContainer" containerID="2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e"
Sep 18 16:08:45.268436 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.268422    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e"} err="failed to get container status \"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e\": rpc error: code = NotFound desc = could not find container \"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e\": container with ID starting with 2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e not found: ID does not exist"
Sep 18 16:08:45.268487 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.268436    2554 scope.go:117] "RemoveContainer" containerID="8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555"
Sep 18 16:08:45.268668 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.268651    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555"} err="failed to get container status \"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555\": rpc error: code = NotFound desc = could not find container \"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555\": container with ID starting with 8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555 not found: ID does not exist"
Sep 18 16:08:45.268722 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.268667    2554 scope.go:117] "RemoveContainer" containerID="c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd"
Sep 18 16:08:45.268839 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.268825    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd"} err="failed to get container status \"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd\": rpc error: code = NotFound desc = could not find container \"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd\": container with ID starting with c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd not found: ID does not exist"
Sep 18 16:08:45.268880 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.268839    2554 scope.go:117] "RemoveContainer" containerID="63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10"
Sep 18 16:08:45.269028 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.269014    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10"} err="failed to get container status \"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10\": rpc error: code = NotFound desc = could not find container \"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10\": container with ID starting with 63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10 not found: ID does not exist"
Sep 18 16:08:45.269091 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.269027    2554 scope.go:117] "RemoveContainer" containerID="f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b"
Sep 18 16:08:45.269257 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.269241    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b"} err="failed to get container status \"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b\": rpc error: code = NotFound desc = could not find container \"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b\": container with ID starting with f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b not found: ID does not exist"
Sep 18 16:08:45.269306 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.269257    2554 scope.go:117] "RemoveContainer" containerID="57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e"
Sep 18 16:08:45.269488 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.269469    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e"} err="failed to get container status \"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e\": rpc error: code = NotFound desc = could not find container \"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e\": container with ID starting with 57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e not found: ID does not exist"
Sep 18 16:08:45.269592 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.269488    2554 scope.go:117] "RemoveContainer" containerID="fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d"
Sep 18 16:08:45.269756 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.269739    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d"} err="failed to get container status \"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d\": rpc error: code = NotFound desc = could not find container \"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d\": container with ID starting with fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d not found: ID does not exist"
Sep 18 16:08:45.269756 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.269754    2554 scope.go:117] "RemoveContainer" containerID="2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e"
Sep 18 16:08:45.269975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.269958    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e"} err="failed to get container status \"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e\": rpc error: code = NotFound desc = could not find container \"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e\": container with ID starting with 2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e not found: ID does not exist"
Sep 18 16:08:45.270017 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.269976    2554 scope.go:117] "RemoveContainer" containerID="8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555"
Sep 18 16:08:45.270184 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.270167    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555"} err="failed to get container status \"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555\": rpc error: code = NotFound desc = could not find container \"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555\": container with ID starting with 8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555 not found: ID does not exist"
Sep 18 16:08:45.270226 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.270185    2554 scope.go:117] "RemoveContainer" containerID="c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd"
Sep 18 16:08:45.270398 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.270382    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd"} err="failed to get container status \"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd\": rpc error: code = NotFound desc = could not find container \"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd\": container with ID starting with c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd not found: ID does not exist"
Sep 18 16:08:45.270398 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.270398    2554 scope.go:117] "RemoveContainer" containerID="63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10"
Sep 18 16:08:45.270637 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.270619    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10"} err="failed to get container status \"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10\": rpc error: code = NotFound desc = could not find container \"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10\": container with ID starting with 63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10 not found: ID does not exist"
Sep 18 16:08:45.270693 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.270636    2554 scope.go:117] "RemoveContainer" containerID="f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b"
Sep 18 16:08:45.270896 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.270865    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b"} err="failed to get container status \"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b\": rpc error: code = NotFound desc = could not find container \"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b\": container with ID starting with f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b not found: ID does not exist"
Sep 18 16:08:45.270896 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.270885    2554 scope.go:117] "RemoveContainer" containerID="57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e"
Sep 18 16:08:45.271465 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.271396    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e"} err="failed to get container status \"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e\": rpc error: code = NotFound desc = could not find container \"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e\": container with ID starting with 57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e not found: ID does not exist"
Sep 18 16:08:45.271597 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.271467    2554 scope.go:117] "RemoveContainer" containerID="fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d"
Sep 18 16:08:45.271825 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.271792    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d"} err="failed to get container status \"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d\": rpc error: code = NotFound desc = could not find container \"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d\": container with ID starting with fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d not found: ID does not exist"
Sep 18 16:08:45.271912 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.271829    2554 scope.go:117] "RemoveContainer" containerID="2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e"
Sep 18 16:08:45.272136 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.272115    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e"} err="failed to get container status \"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e\": rpc error: code = NotFound desc = could not find container \"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e\": container with ID starting with 2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e not found: ID does not exist"
Sep 18 16:08:45.272136 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.272135    2554 scope.go:117] "RemoveContainer" containerID="8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555"
Sep 18 16:08:45.272406 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.272384    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555"} err="failed to get container status \"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555\": rpc error: code = NotFound desc = could not find container \"8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555\": container with ID starting with 8119283bcedecd6744f14f08157e5db357013ca52c7a34ea43b39dbf5eee0555 not found: ID does not exist"
Sep 18 16:08:45.272406 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.272405    2554 scope.go:117] "RemoveContainer" containerID="c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd"
Sep 18 16:08:45.272675 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.272659    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd"} err="failed to get container status \"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd\": rpc error: code = NotFound desc = could not find container \"c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd\": container with ID starting with c54fe3c80686f1367be70b561b22c834ca131e28e522b9422af96160fe7dc1cd not found: ID does not exist"
Sep 18 16:08:45.272738 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.272676    2554 scope.go:117] "RemoveContainer" containerID="63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10"
Sep 18 16:08:45.272917 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.272901    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10"} err="failed to get container status \"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10\": rpc error: code = NotFound desc = could not find container \"63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10\": container with ID starting with 63ac0ac0ee4c58a5dd4e6103d1d7356c0b29fdabdcfdbc22ba7454d45957ca10 not found: ID does not exist"
Sep 18 16:08:45.272977 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.272916    2554 scope.go:117] "RemoveContainer" containerID="f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b"
Sep 18 16:08:45.273174 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.273150    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b"} err="failed to get container status \"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b\": rpc error: code = NotFound desc = could not find container \"f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b\": container with ID starting with f133fb816578d0fbf1d78ab2c0b43b375c62206ad0b7b7a25cc2310eb6a7781b not found: ID does not exist"
Sep 18 16:08:45.273174 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.273174    2554 scope.go:117] "RemoveContainer" containerID="57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e"
Sep 18 16:08:45.273396 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.273375    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e"} err="failed to get container status \"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e\": rpc error: code = NotFound desc = could not find container \"57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e\": container with ID starting with 57780c9d4cef17edc1cf28f058539ad6a94e1ba16f68a8c0a01b2816d3cf508e not found: ID does not exist"
Sep 18 16:08:45.273440 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.273398    2554 scope.go:117] "RemoveContainer" containerID="fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d"
Sep 18 16:08:45.273706 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.273686    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d"} err="failed to get container status \"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d\": rpc error: code = NotFound desc = could not find container \"fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d\": container with ID starting with fdc93b52415a8156f34947731b771f92e027f84b5304ea8ee640f430c7eeff4d not found: ID does not exist"
Sep 18 16:08:45.273766 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.273709    2554 scope.go:117] "RemoveContainer" containerID="2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e"
Sep 18 16:08:45.273942 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.273926    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e"} err="failed to get container status \"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e\": rpc error: code = NotFound desc = could not find container \"2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e\": container with ID starting with 2254397f655ccaab8bcbbe9f7f12d3611b96d8c7fbd69d9facd9a4921efb7f3e not found: ID does not exist"
Sep 18 16:08:45.275916 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.275898    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" path="/var/lib/kubelet/pods/cc9ffc5c-f0cd-491f-919d-6e7182b61d86/volumes"
Sep 18 16:08:45.276364 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276350    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:08:45.276636 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276623    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="kube-rbac-proxy-web"
Sep 18 16:08:45.276680 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276638    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="kube-rbac-proxy-web"
Sep 18 16:08:45.276680 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276647    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7" containerName="registry"
Sep 18 16:08:45.276680 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276652    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7" containerName="registry"
Sep 18 16:08:45.276680 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276666    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="prometheus"
Sep 18 16:08:45.276680 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276671    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="prometheus"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276682    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="thanos-sidecar"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276688    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="thanos-sidecar"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276695    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="init-config-reloader"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276700    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="init-config-reloader"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276709    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="kube-rbac-proxy-thanos"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276714    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="kube-rbac-proxy-thanos"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276721    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="config-reloader"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276726    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="config-reloader"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276731    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="kube-rbac-proxy"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276735    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="kube-rbac-proxy"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276775    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="prometheus"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276782    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="kube-rbac-proxy-web"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276788    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="config-reloader"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276795    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c272d7f1-2c0a-42df-8fbe-e8c8606bbfb7" containerName="registry"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276800    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="thanos-sidecar"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276805    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="kube-rbac-proxy"
Sep 18 16:08:45.276817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.276812    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9ffc5c-f0cd-491f-919d-6e7182b61d86" containerName="kube-rbac-proxy-thanos"
Sep 18 16:08:45.281207 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.281191    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.283678 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.283660    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 16:08:45.283772 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.283662    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rnhrt\""
Sep 18 16:08:45.283979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.283966    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 16:08:45.288955 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.286192    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 16:08:45.288955 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.286541    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 16:08:45.288955 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.286822    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 16:08:45.288955 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.287038    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cduku6kld38gr\""
Sep 18 16:08:45.288955 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.287278    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 16:08:45.288955 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.287895    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 16:08:45.288955 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.288139    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 16:08:45.288955 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.288618    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 16:08:45.288955 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.288668    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 16:08:45.289490 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.288981    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 16:08:45.290008 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.289978    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 16:08:45.291462 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.290859    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 16:08:45.293323 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.293110    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 16:08:45.295565 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.295544    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:08:45.295970 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.295950    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 16:08:45.423195 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423155    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423195 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423192    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc31abd9-b61b-4620-a13e-1339d175dae4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423392 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423216    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423392 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423253    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423392 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423311    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-config\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423392 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423340    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423392 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423359    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423392 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423384    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423603 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423408    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423603 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423428    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc31abd9-b61b-4620-a13e-1339d175dae4-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423603 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423445    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423603 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423491    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423603 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423549    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423603 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423579    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423603 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423596    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423814 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423619    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423814 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423645    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423814 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423659    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc31abd9-b61b-4620-a13e-1339d175dae4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423814 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423678    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.423814 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.423734    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stsd8\" (UniqueName: \"kubernetes.io/projected/fc31abd9-b61b-4620-a13e-1339d175dae4-kube-api-access-stsd8\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.524149 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.524059    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stsd8\" (UniqueName: \"kubernetes.io/projected/fc31abd9-b61b-4620-a13e-1339d175dae4-kube-api-access-stsd8\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.524149 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.524099    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.524149 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.524126    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc31abd9-b61b-4620-a13e-1339d175dae4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.524437 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.524239    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.524437 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.524267    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.524437 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.524287    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-config\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.524437 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.524311    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.524437 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.524335    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.524437 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.524362    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.524437 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.524402    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.525108 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.524443    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc31abd9-b61b-4620-a13e-1339d175dae4-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.525108 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.524467    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.525108 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.524946    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.525497 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.525391    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.525497 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.525444    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.525497 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.525485    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.525675 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.525548    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.525675 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.525578    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.525675 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.525608    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.525675 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.525651    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.525876 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.525675    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc31abd9-b61b-4620-a13e-1339d175dae4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.525876 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.525715    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.526557 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.526224    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.527385 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.527237    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.527729 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.527497    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc31abd9-b61b-4620-a13e-1339d175dae4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.527729 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.527538    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.528006 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.527981    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.528314 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.528283    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.528447 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.528424    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.528852 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.528828    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.529117 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.529091    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.529207 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.529131    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc31abd9-b61b-4620-a13e-1339d175dae4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.529388 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.529367    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-config\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.529880 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.529863    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc31abd9-b61b-4620-a13e-1339d175dae4-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.530234 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.530214    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc31abd9-b61b-4620-a13e-1339d175dae4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.530591 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.530573    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.530648 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.530594    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.530948 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.530919    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.531344 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.531324    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc31abd9-b61b-4620-a13e-1339d175dae4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.539965 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.539935    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stsd8\" (UniqueName: \"kubernetes.io/projected/fc31abd9-b61b-4620-a13e-1339d175dae4-kube-api-access-stsd8\") pod \"prometheus-k8s-0\" (UID: \"fc31abd9-b61b-4620-a13e-1339d175dae4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.603102 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.603059    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:08:45.889477 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:45.889454    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 16:08:45.891861 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:08:45.891837    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc31abd9_b61b_4620_a13e_1339d175dae4.slice/crio-8d51a803de6b06095d3aad34033073d35ba0d9a6b10dfae397afacca0a4a2a77 WatchSource:0}: Error finding container 8d51a803de6b06095d3aad34033073d35ba0d9a6b10dfae397afacca0a4a2a77: Status 404 returned error can't find the container with id 8d51a803de6b06095d3aad34033073d35ba0d9a6b10dfae397afacca0a4a2a77
Sep 18 16:08:46.219404 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:46.219366    2554 generic.go:356] "Generic (PLEG): container finished" podID="fc31abd9-b61b-4620-a13e-1339d175dae4" containerID="536d7c4c5edd2f65a607accbcb9bbf4bf4d61cf178ac27d66fbe7011a485a1a7" exitCode=0
Sep 18 16:08:46.219596 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:46.219437    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc31abd9-b61b-4620-a13e-1339d175dae4","Type":"ContainerDied","Data":"536d7c4c5edd2f65a607accbcb9bbf4bf4d61cf178ac27d66fbe7011a485a1a7"}
Sep 18 16:08:46.219596 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:46.219497    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc31abd9-b61b-4620-a13e-1339d175dae4","Type":"ContainerStarted","Data":"8d51a803de6b06095d3aad34033073d35ba0d9a6b10dfae397afacca0a4a2a77"}
Sep 18 16:08:47.227295 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:47.227261    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc31abd9-b61b-4620-a13e-1339d175dae4","Type":"ContainerStarted","Data":"35199bce2799aea5d1286bd72dd496498b158819a8b8c1af0e9b8165f92f98f6"}
Sep 18 16:08:47.227697 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:47.227324    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc31abd9-b61b-4620-a13e-1339d175dae4","Type":"ContainerStarted","Data":"6b5369b19fd686ae20177ae0bd7c0046ad17200fa64a675224d902adb6e9812b"}
Sep 18 16:08:47.227697 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:47.227358    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc31abd9-b61b-4620-a13e-1339d175dae4","Type":"ContainerStarted","Data":"4155fbae85a1e470c8a01b6750a22d03cb57d3c1335637dc57df093f115554f0"}
Sep 18 16:08:47.227697 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:47.227382    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc31abd9-b61b-4620-a13e-1339d175dae4","Type":"ContainerStarted","Data":"687bfd2527389005094131821c519ebb54961fbb975e6691fb67e62e806116bb"}
Sep 18 16:08:47.227697 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:47.227403    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc31abd9-b61b-4620-a13e-1339d175dae4","Type":"ContainerStarted","Data":"f942c6aedabc7da5df333d65672c06951be862fa0c502b80789d611cdf94d9c1"}
Sep 18 16:08:47.227697 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:47.227451    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc31abd9-b61b-4620-a13e-1339d175dae4","Type":"ContainerStarted","Data":"d36a350d4fa5213043a541a6520b1ff6f4fab07d8c834e38294936cbeb752314"}
Sep 18 16:08:47.260884 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:47.260823    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.242079348 podStartE2EDuration="2.26080126s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.721912ms" isStatefulPod=false podCreationTimestamp="2026-09-18 16:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:08:47.259217106 +0000 UTC m=+222.206715881" watchObservedRunningTime="2026-09-18 16:08:47.26080126 +0000 UTC m=+222.208300036"
Sep 18 16:08:50.603358 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:08:50.603316    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:09:45.603697 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:09:45.603660    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:09:45.619540 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:09:45.619499    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:09:46.466434 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:09:46.466409    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 16:10:07.161866 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:10:07.161838    2554 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 16:11:23.609463 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.609380    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"]
Sep 18 16:11:23.612757 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.612738    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:23.615404 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.615382    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 16:11:23.615754 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.615736    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 16:11:23.615857 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.615827    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 16:11:23.616207 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.616193    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 16:11:23.616278 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.616194    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mvfv2\""
Sep 18 16:11:23.630063 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.630040    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"]
Sep 18 16:11:23.758947 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.758910    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64d4aca4-7571-4c30-a7c8-b1747b3635b6-manifests\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57\" (UID: \"64d4aca4-7571-4c30-a7c8-b1747b3635b6\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:23.758947 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.758943    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64d4aca4-7571-4c30-a7c8-b1747b3635b6-tmp\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57\" (UID: \"64d4aca4-7571-4c30-a7c8-b1747b3635b6\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:23.759140 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.758973    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64d4aca4-7571-4c30-a7c8-b1747b3635b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57\" (UID: \"64d4aca4-7571-4c30-a7c8-b1747b3635b6\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:23.759140 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.759022    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdlr5\" (UniqueName: \"kubernetes.io/projected/64d4aca4-7571-4c30-a7c8-b1747b3635b6-kube-api-access-jdlr5\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57\" (UID: \"64d4aca4-7571-4c30-a7c8-b1747b3635b6\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:23.759140 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.759066    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64d4aca4-7571-4c30-a7c8-b1747b3635b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57\" (UID: \"64d4aca4-7571-4c30-a7c8-b1747b3635b6\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:23.860243 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.860161    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64d4aca4-7571-4c30-a7c8-b1747b3635b6-manifests\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57\" (UID: \"64d4aca4-7571-4c30-a7c8-b1747b3635b6\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:23.860243 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.860199    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64d4aca4-7571-4c30-a7c8-b1747b3635b6-tmp\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57\" (UID: \"64d4aca4-7571-4c30-a7c8-b1747b3635b6\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:23.860243 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.860226    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64d4aca4-7571-4c30-a7c8-b1747b3635b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57\" (UID: \"64d4aca4-7571-4c30-a7c8-b1747b3635b6\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:23.860517 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.860250    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdlr5\" (UniqueName: \"kubernetes.io/projected/64d4aca4-7571-4c30-a7c8-b1747b3635b6-kube-api-access-jdlr5\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57\" (UID: \"64d4aca4-7571-4c30-a7c8-b1747b3635b6\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:23.860517 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.860291    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64d4aca4-7571-4c30-a7c8-b1747b3635b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57\" (UID: \"64d4aca4-7571-4c30-a7c8-b1747b3635b6\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:23.860665 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.860577    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64d4aca4-7571-4c30-a7c8-b1747b3635b6-manifests\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57\" (UID: \"64d4aca4-7571-4c30-a7c8-b1747b3635b6\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:23.862628 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.862607    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64d4aca4-7571-4c30-a7c8-b1747b3635b6-tmp\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57\" (UID: \"64d4aca4-7571-4c30-a7c8-b1747b3635b6\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:23.862849 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.862832    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64d4aca4-7571-4c30-a7c8-b1747b3635b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57\" (UID: \"64d4aca4-7571-4c30-a7c8-b1747b3635b6\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:23.862984 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.862965    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64d4aca4-7571-4c30-a7c8-b1747b3635b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57\" (UID: \"64d4aca4-7571-4c30-a7c8-b1747b3635b6\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:23.870425 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.870406    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdlr5\" (UniqueName: \"kubernetes.io/projected/64d4aca4-7571-4c30-a7c8-b1747b3635b6-kube-api-access-jdlr5\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57\" (UID: \"64d4aca4-7571-4c30-a7c8-b1747b3635b6\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:23.922296 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:23.922266    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:24.208246 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:24.208215    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"]
Sep 18 16:11:24.211713 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:11:24.211686    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d4aca4_7571_4c30_a7c8_b1747b3635b6.slice/crio-300140ddf00c24f1830f909b3a1ccc553b92a0e92776f4e14b0e5fd2dccf7928 WatchSource:0}: Error finding container 300140ddf00c24f1830f909b3a1ccc553b92a0e92776f4e14b0e5fd2dccf7928: Status 404 returned error can't find the container with id 300140ddf00c24f1830f909b3a1ccc553b92a0e92776f4e14b0e5fd2dccf7928
Sep 18 16:11:24.213546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:24.213510    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:11:24.820919 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:24.820870    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" event={"ID":"64d4aca4-7571-4c30-a7c8-b1747b3635b6","Type":"ContainerStarted","Data":"300140ddf00c24f1830f909b3a1ccc553b92a0e92776f4e14b0e5fd2dccf7928"}
Sep 18 16:11:27.838377 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:27.838341    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" event={"ID":"64d4aca4-7571-4c30-a7c8-b1747b3635b6","Type":"ContainerStarted","Data":"899b1099afacfbd1390467af0d02cf85855bcf162b786bc47cf67b3e74d4151e"}
Sep 18 16:11:28.844281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:28.844195    2554 generic.go:356] "Generic (PLEG): container finished" podID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerID="899b1099afacfbd1390467af0d02cf85855bcf162b786bc47cf67b3e74d4151e" exitCode=0
Sep 18 16:11:28.844281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:28.844250    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" event={"ID":"64d4aca4-7571-4c30-a7c8-b1747b3635b6","Type":"ContainerDied","Data":"899b1099afacfbd1390467af0d02cf85855bcf162b786bc47cf67b3e74d4151e"}
Sep 18 16:11:29.850794 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:29.850760    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" event={"ID":"64d4aca4-7571-4c30-a7c8-b1747b3635b6","Type":"ContainerStarted","Data":"b1ac7815ed22d38535e520a06fdad108186adfd4f8ad23fd553abb4e925c4e77"}
Sep 18 16:11:29.851202 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:29.850874    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:29.877007 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:29.876957    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podStartSLOduration=2.216208523 podStartE2EDuration="6.876941853s" totalImagesPullingTime="3.572778968s" totalInitContainerRuntime="1.087954362s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:11:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:11:29.874830062 +0000 UTC m=+384.822328833" watchObservedRunningTime="2026-09-18 16:11:29.876941853 +0000 UTC m=+384.824440626"
Sep 18 16:11:40.856798 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:40.856765    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:11:41.407608 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.407573    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"]
Sep 18 16:11:41.417263 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.417238    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:11:41.420201 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.420174    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"]
Sep 18 16:11:41.420469 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.420449    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 16:11:41.420619 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.420578    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 16:11:41.420689 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.420553    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ln5jh\""
Sep 18 16:11:41.420749 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.420562    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 16:11:41.420749 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.420597    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 16:11:41.420852 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.420459    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:11:41.498537 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.498497    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5t8h\" (UniqueName: \"kubernetes.io/projected/720642cf-540a-42d3-ab46-487cf5a95e72-kube-api-access-b5t8h\") pod \"lws-controller-manager-6847d847f7-2hft6\" (UID: \"720642cf-540a-42d3-ab46-487cf5a95e72\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:11:41.498699 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.498550    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/720642cf-540a-42d3-ab46-487cf5a95e72-manager-config\") pod \"lws-controller-manager-6847d847f7-2hft6\" (UID: \"720642cf-540a-42d3-ab46-487cf5a95e72\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:11:41.498699 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.498613    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/720642cf-540a-42d3-ab46-487cf5a95e72-metrics-cert\") pod \"lws-controller-manager-6847d847f7-2hft6\" (UID: \"720642cf-540a-42d3-ab46-487cf5a95e72\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:11:41.498699 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.498651    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/720642cf-540a-42d3-ab46-487cf5a95e72-cert\") pod \"lws-controller-manager-6847d847f7-2hft6\" (UID: \"720642cf-540a-42d3-ab46-487cf5a95e72\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:11:41.599033 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.598989    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5t8h\" (UniqueName: \"kubernetes.io/projected/720642cf-540a-42d3-ab46-487cf5a95e72-kube-api-access-b5t8h\") pod \"lws-controller-manager-6847d847f7-2hft6\" (UID: \"720642cf-540a-42d3-ab46-487cf5a95e72\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:11:41.599235 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.599044    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/720642cf-540a-42d3-ab46-487cf5a95e72-manager-config\") pod \"lws-controller-manager-6847d847f7-2hft6\" (UID: \"720642cf-540a-42d3-ab46-487cf5a95e72\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:11:41.599235 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.599074    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/720642cf-540a-42d3-ab46-487cf5a95e72-metrics-cert\") pod \"lws-controller-manager-6847d847f7-2hft6\" (UID: \"720642cf-540a-42d3-ab46-487cf5a95e72\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:11:41.599353 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.599235    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/720642cf-540a-42d3-ab46-487cf5a95e72-cert\") pod \"lws-controller-manager-6847d847f7-2hft6\" (UID: \"720642cf-540a-42d3-ab46-487cf5a95e72\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:11:41.599794 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.599771    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/720642cf-540a-42d3-ab46-487cf5a95e72-manager-config\") pod \"lws-controller-manager-6847d847f7-2hft6\" (UID: \"720642cf-540a-42d3-ab46-487cf5a95e72\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:11:41.601591 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.601563    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/720642cf-540a-42d3-ab46-487cf5a95e72-metrics-cert\") pod \"lws-controller-manager-6847d847f7-2hft6\" (UID: \"720642cf-540a-42d3-ab46-487cf5a95e72\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:11:41.601675 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.601652    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/720642cf-540a-42d3-ab46-487cf5a95e72-cert\") pod \"lws-controller-manager-6847d847f7-2hft6\" (UID: \"720642cf-540a-42d3-ab46-487cf5a95e72\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:11:41.606966 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.606948    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5t8h\" (UniqueName: \"kubernetes.io/projected/720642cf-540a-42d3-ab46-487cf5a95e72-kube-api-access-b5t8h\") pod \"lws-controller-manager-6847d847f7-2hft6\" (UID: \"720642cf-540a-42d3-ab46-487cf5a95e72\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:11:41.727123 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.727099    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:11:41.997510 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:41.997439    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"]
Sep 18 16:11:42.000593 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:11:42.000554    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720642cf_540a_42d3_ab46_487cf5a95e72.slice/crio-12b2e5ab569e2feb245c6cf4b7ab995c3e1e795106f5612a3014e3ff924dc1e0 WatchSource:0}: Error finding container 12b2e5ab569e2feb245c6cf4b7ab995c3e1e795106f5612a3014e3ff924dc1e0: Status 404 returned error can't find the container with id 12b2e5ab569e2feb245c6cf4b7ab995c3e1e795106f5612a3014e3ff924dc1e0
Sep 18 16:11:42.907547 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:42.907459    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6" event={"ID":"720642cf-540a-42d3-ab46-487cf5a95e72","Type":"ContainerStarted","Data":"12b2e5ab569e2feb245c6cf4b7ab995c3e1e795106f5612a3014e3ff924dc1e0"}
Sep 18 16:11:44.918658 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:44.918567    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6" event={"ID":"720642cf-540a-42d3-ab46-487cf5a95e72","Type":"ContainerStarted","Data":"9921139325e7f8035cb3368ff0a010aac348cb93f1f6a9148ad976c4643013e9"}
Sep 18 16:11:44.919011 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:44.918685    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:11:44.937851 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:44.937810    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6" podStartSLOduration=1.737629433 podStartE2EDuration="3.937796132s" totalImagesPullingTime="2.200166699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:11:44.936057913 +0000 UTC m=+399.883556689" watchObservedRunningTime="2026-09-18 16:11:44.937796132 +0000 UTC m=+399.885294905"
Sep 18 16:11:53.681177 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.681141    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk"]
Sep 18 16:11:53.684256 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.684240    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk"
Sep 18 16:11:53.688649 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.688626    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 16:11:53.689405 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.689367    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 16:11:53.689405 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.689381    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 16:11:53.689659 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.689402    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6z2h8\""
Sep 18 16:11:53.692017 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.691998    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 16:11:53.716775 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.716748    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk"]
Sep 18 16:11:53.800237 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.800206    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d148fb58-8dd0-4d6f-b2a1-35a54628b0ad-tmp\") pod \"kube-auth-proxy-f97b6d9fd-kq8jk\" (UID: \"d148fb58-8dd0-4d6f-b2a1-35a54628b0ad\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk"
Sep 18 16:11:53.800395 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.800257    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d148fb58-8dd0-4d6f-b2a1-35a54628b0ad-tls-certs\") pod \"kube-auth-proxy-f97b6d9fd-kq8jk\" (UID: \"d148fb58-8dd0-4d6f-b2a1-35a54628b0ad\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk"
Sep 18 16:11:53.800395 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.800279    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l6xl\" (UniqueName: \"kubernetes.io/projected/d148fb58-8dd0-4d6f-b2a1-35a54628b0ad-kube-api-access-6l6xl\") pod \"kube-auth-proxy-f97b6d9fd-kq8jk\" (UID: \"d148fb58-8dd0-4d6f-b2a1-35a54628b0ad\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk"
Sep 18 16:11:53.900815 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.900779    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d148fb58-8dd0-4d6f-b2a1-35a54628b0ad-tmp\") pod \"kube-auth-proxy-f97b6d9fd-kq8jk\" (UID: \"d148fb58-8dd0-4d6f-b2a1-35a54628b0ad\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk"
Sep 18 16:11:53.900972 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.900825    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d148fb58-8dd0-4d6f-b2a1-35a54628b0ad-tls-certs\") pod \"kube-auth-proxy-f97b6d9fd-kq8jk\" (UID: \"d148fb58-8dd0-4d6f-b2a1-35a54628b0ad\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk"
Sep 18 16:11:53.900972 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.900846    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l6xl\" (UniqueName: \"kubernetes.io/projected/d148fb58-8dd0-4d6f-b2a1-35a54628b0ad-kube-api-access-6l6xl\") pod \"kube-auth-proxy-f97b6d9fd-kq8jk\" (UID: \"d148fb58-8dd0-4d6f-b2a1-35a54628b0ad\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk"
Sep 18 16:11:53.903293 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.903264    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d148fb58-8dd0-4d6f-b2a1-35a54628b0ad-tmp\") pod \"kube-auth-proxy-f97b6d9fd-kq8jk\" (UID: \"d148fb58-8dd0-4d6f-b2a1-35a54628b0ad\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk"
Sep 18 16:11:53.903430 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.903380    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d148fb58-8dd0-4d6f-b2a1-35a54628b0ad-tls-certs\") pod \"kube-auth-proxy-f97b6d9fd-kq8jk\" (UID: \"d148fb58-8dd0-4d6f-b2a1-35a54628b0ad\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk"
Sep 18 16:11:53.917908 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.917878    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l6xl\" (UniqueName: \"kubernetes.io/projected/d148fb58-8dd0-4d6f-b2a1-35a54628b0ad-kube-api-access-6l6xl\") pod \"kube-auth-proxy-f97b6d9fd-kq8jk\" (UID: \"d148fb58-8dd0-4d6f-b2a1-35a54628b0ad\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk"
Sep 18 16:11:53.993699 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:53.993609    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk"
Sep 18 16:11:54.270135 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:54.270058    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk"]
Sep 18 16:11:54.273090 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:11:54.273061    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd148fb58_8dd0_4d6f_b2a1_35a54628b0ad.slice/crio-f0c133e0a640fd92cb6c113cb34f0ab4f40bf3144235a2a217c622c72abb5261 WatchSource:0}: Error finding container f0c133e0a640fd92cb6c113cb34f0ab4f40bf3144235a2a217c622c72abb5261: Status 404 returned error can't find the container with id f0c133e0a640fd92cb6c113cb34f0ab4f40bf3144235a2a217c622c72abb5261
Sep 18 16:11:54.960586 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:54.960545    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk" event={"ID":"d148fb58-8dd0-4d6f-b2a1-35a54628b0ad","Type":"ContainerStarted","Data":"f0c133e0a640fd92cb6c113cb34f0ab4f40bf3144235a2a217c622c72abb5261"}
Sep 18 16:11:55.925333 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:55.925303    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:11:57.978347 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:57.978310    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk" event={"ID":"d148fb58-8dd0-4d6f-b2a1-35a54628b0ad","Type":"ContainerStarted","Data":"ec72b4548940105eaf37c5a952e91142b90e772dbcc2b918dbc765efa13ed30e"}
Sep 18 16:11:58.000667 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:11:58.000610    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-kq8jk" podStartSLOduration=1.9886065849999999 podStartE2EDuration="5.000595504s" totalImagesPullingTime="3.011988919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:11:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:11:57.999415254 +0000 UTC m=+412.946914031" watchObservedRunningTime="2026-09-18 16:11:58.000595504 +0000 UTC m=+412.948094278"
Sep 18 16:13:18.427552 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:18.427504    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"]
Sep 18 16:13:18.429817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:18.429801    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:13:18.434367 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:18.434341    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 16:13:18.434636 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:18.434618    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-hqfkr\""
Sep 18 16:13:18.437118 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:18.437100    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 18 16:13:18.438962 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:18.438945    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 16:13:18.448773 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:18.448750    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"]
Sep 18 16:13:18.595244 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:18.595217    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q8k8\" (UniqueName: \"kubernetes.io/projected/ad3586f9-9035-40c1-bf26-143eeb8ff319-kube-api-access-2q8k8\") pod \"dns-operator-controller-manager-7b84cd7c9c-vrmv8\" (UID: \"ad3586f9-9035-40c1-bf26-143eeb8ff319\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:13:18.696391 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:18.696316    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q8k8\" (UniqueName: \"kubernetes.io/projected/ad3586f9-9035-40c1-bf26-143eeb8ff319-kube-api-access-2q8k8\") pod \"dns-operator-controller-manager-7b84cd7c9c-vrmv8\" (UID: \"ad3586f9-9035-40c1-bf26-143eeb8ff319\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:13:18.706810 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:18.706786    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q8k8\" (UniqueName: \"kubernetes.io/projected/ad3586f9-9035-40c1-bf26-143eeb8ff319-kube-api-access-2q8k8\") pod \"dns-operator-controller-manager-7b84cd7c9c-vrmv8\" (UID: \"ad3586f9-9035-40c1-bf26-143eeb8ff319\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:13:18.739670 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:18.739645    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:13:19.007925 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:19.007853    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"]
Sep 18 16:13:19.010157 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:13:19.010124    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3586f9_9035_40c1_bf26_143eeb8ff319.slice/crio-235e15cc5aa9d1b901a598dcc1547e71eed25d01d68cae8459471c3124aed320 WatchSource:0}: Error finding container 235e15cc5aa9d1b901a598dcc1547e71eed25d01d68cae8459471c3124aed320: Status 404 returned error can't find the container with id 235e15cc5aa9d1b901a598dcc1547e71eed25d01d68cae8459471c3124aed320
Sep 18 16:13:19.319201 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:19.319129    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8" event={"ID":"ad3586f9-9035-40c1-bf26-143eeb8ff319","Type":"ContainerStarted","Data":"235e15cc5aa9d1b901a598dcc1547e71eed25d01d68cae8459471c3124aed320"}
Sep 18 16:13:22.341760 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:22.341720    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8" event={"ID":"ad3586f9-9035-40c1-bf26-143eeb8ff319","Type":"ContainerStarted","Data":"8a197b6472763abacc1c82523b1a45616f445b38623fc8ae8fbb86a61f162311"}
Sep 18 16:13:22.342162 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:22.341863    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:13:22.363267 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:22.363222    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8" podStartSLOduration=1.800411246 podStartE2EDuration="4.36320754s" totalImagesPullingTime="2.562796294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:13:22.361320563 +0000 UTC m=+497.308819338" watchObservedRunningTime="2026-09-18 16:13:22.36320754 +0000 UTC m=+497.310706313"
Sep 18 16:13:31.404537 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:31.404503    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9"]
Sep 18 16:13:31.407231 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:31.407213    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9"
Sep 18 16:13:31.410648 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:31.410623    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 18 16:13:31.410648 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:31.410639    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dc6ds\""
Sep 18 16:13:31.410806 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:31.410623    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 18 16:13:31.420017 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:31.419996    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9"]
Sep 18 16:13:31.493833 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:31.493804    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bae5c766-ff8d-4a94-a9d3-0f091dbba057-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-mbck9\" (UID: \"bae5c766-ff8d-4a94-a9d3-0f091dbba057\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9"
Sep 18 16:13:31.493965 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:31.493851    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bae5c766-ff8d-4a94-a9d3-0f091dbba057-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-mbck9\" (UID: \"bae5c766-ff8d-4a94-a9d3-0f091dbba057\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9"
Sep 18 16:13:31.493965 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:31.493908    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppgnp\" (UniqueName: \"kubernetes.io/projected/bae5c766-ff8d-4a94-a9d3-0f091dbba057-kube-api-access-ppgnp\") pod \"kuadrant-console-plugin-fbdd4f445-mbck9\" (UID: \"bae5c766-ff8d-4a94-a9d3-0f091dbba057\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9"
Sep 18 16:13:31.595095 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:31.595064    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bae5c766-ff8d-4a94-a9d3-0f091dbba057-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-mbck9\" (UID: \"bae5c766-ff8d-4a94-a9d3-0f091dbba057\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9"
Sep 18 16:13:31.595276 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:31.595129    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppgnp\" (UniqueName: \"kubernetes.io/projected/bae5c766-ff8d-4a94-a9d3-0f091dbba057-kube-api-access-ppgnp\") pod \"kuadrant-console-plugin-fbdd4f445-mbck9\" (UID: \"bae5c766-ff8d-4a94-a9d3-0f091dbba057\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9"
Sep 18 16:13:31.595276 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:31.595153    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bae5c766-ff8d-4a94-a9d3-0f091dbba057-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-mbck9\" (UID: \"bae5c766-ff8d-4a94-a9d3-0f091dbba057\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9"
Sep 18 16:13:31.595842 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:31.595822    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bae5c766-ff8d-4a94-a9d3-0f091dbba057-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-mbck9\" (UID: \"bae5c766-ff8d-4a94-a9d3-0f091dbba057\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9"
Sep 18 16:13:31.597552 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:31.597512    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bae5c766-ff8d-4a94-a9d3-0f091dbba057-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-mbck9\" (UID: \"bae5c766-ff8d-4a94-a9d3-0f091dbba057\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9"
Sep 18 16:13:31.609920 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:31.609889    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppgnp\" (UniqueName: \"kubernetes.io/projected/bae5c766-ff8d-4a94-a9d3-0f091dbba057-kube-api-access-ppgnp\") pod \"kuadrant-console-plugin-fbdd4f445-mbck9\" (UID: \"bae5c766-ff8d-4a94-a9d3-0f091dbba057\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9"
Sep 18 16:13:31.734515 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:31.734495    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9"
Sep 18 16:13:32.010943 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:32.010874    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9"]
Sep 18 16:13:32.013139 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:13:32.013112    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbae5c766_ff8d_4a94_a9d3_0f091dbba057.slice/crio-d9a34b355d93b1b00121e8c8f0ad2532f33a7da65d40f000612d9d42d5ffb84e WatchSource:0}: Error finding container d9a34b355d93b1b00121e8c8f0ad2532f33a7da65d40f000612d9d42d5ffb84e: Status 404 returned error can't find the container with id d9a34b355d93b1b00121e8c8f0ad2532f33a7da65d40f000612d9d42d5ffb84e
Sep 18 16:13:32.390001 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:32.389933    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9" event={"ID":"bae5c766-ff8d-4a94-a9d3-0f091dbba057","Type":"ContainerStarted","Data":"d9a34b355d93b1b00121e8c8f0ad2532f33a7da65d40f000612d9d42d5ffb84e"}
Sep 18 16:13:33.349054 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:33.348959    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:13:37.415950 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:37.415915    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9" event={"ID":"bae5c766-ff8d-4a94-a9d3-0f091dbba057","Type":"ContainerStarted","Data":"b05bd5cd35db14396305b2d27e9d121eb93c7a601385c6517c59c8d7f291048d"}
Sep 18 16:13:37.434517 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:37.434465    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mbck9" podStartSLOduration=1.73174274 podStartE2EDuration="6.434452101s" totalImagesPullingTime="4.702709361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:13:37.433141455 +0000 UTC m=+512.380640231" watchObservedRunningTime="2026-09-18 16:13:37.434452101 +0000 UTC m=+512.381950907"
Sep 18 16:13:58.843983 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.843904    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq"]
Sep 18 16:13:58.861056 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.861015    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq"
Sep 18 16:13:58.862963 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.862929    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"]
Sep 18 16:13:58.863719 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.863701    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 16:13:58.863847 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.863752    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 16:13:58.864104 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.864084    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 16:13:58.864739 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.864720    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 16:13:58.866290 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.866272    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.867279 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.867260    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq"]
Sep 18 16:13:58.882850 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.882831    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 18 16:13:58.883002 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.882987    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 18 16:13:58.883081 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.883068    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 18 16:13:58.883129 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.883085    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 18 16:13:58.890422 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.890406    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/afd25d28-d917-4a77-bb0a-2003215eb405-hub\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.890476 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.890444    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4gxk\" (UniqueName: \"kubernetes.io/projected/afd25d28-d917-4a77-bb0a-2003215eb405-kube-api-access-n4gxk\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.890476 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.890461    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctlhz\" (UniqueName: \"kubernetes.io/projected/9701a90e-fb16-49b2-b66a-dc53637da155-kube-api-access-ctlhz\") pod \"klusterlet-addon-workmgr-bbcb9d47c-b7tcq\" (UID: \"9701a90e-fb16-49b2-b66a-dc53637da155\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq"
Sep 18 16:13:58.890567 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.890477    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/afd25d28-d917-4a77-bb0a-2003215eb405-ca\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.890567 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.890556    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9701a90e-fb16-49b2-b66a-dc53637da155-klusterlet-config\") pod \"klusterlet-addon-workmgr-bbcb9d47c-b7tcq\" (UID: \"9701a90e-fb16-49b2-b66a-dc53637da155\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq"
Sep 18 16:13:58.890633 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.890606    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afd25d28-d917-4a77-bb0a-2003215eb405-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.890667 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.890639    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/afd25d28-d917-4a77-bb0a-2003215eb405-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.890667 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.890658    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9701a90e-fb16-49b2-b66a-dc53637da155-tmp\") pod \"klusterlet-addon-workmgr-bbcb9d47c-b7tcq\" (UID: \"9701a90e-fb16-49b2-b66a-dc53637da155\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq"
Sep 18 16:13:58.890728 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.890677    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd25d28-d917-4a77-bb0a-2003215eb405-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.906639 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.906615    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"]
Sep 18 16:13:58.991027 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.991001    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/afd25d28-d917-4a77-bb0a-2003215eb405-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.991185 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.991032    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9701a90e-fb16-49b2-b66a-dc53637da155-tmp\") pod \"klusterlet-addon-workmgr-bbcb9d47c-b7tcq\" (UID: \"9701a90e-fb16-49b2-b66a-dc53637da155\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq"
Sep 18 16:13:58.991185 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.991053    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd25d28-d917-4a77-bb0a-2003215eb405-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.991185 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.991086    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/afd25d28-d917-4a77-bb0a-2003215eb405-hub\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.991185 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.991134    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4gxk\" (UniqueName: \"kubernetes.io/projected/afd25d28-d917-4a77-bb0a-2003215eb405-kube-api-access-n4gxk\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.991185 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.991159    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctlhz\" (UniqueName: \"kubernetes.io/projected/9701a90e-fb16-49b2-b66a-dc53637da155-kube-api-access-ctlhz\") pod \"klusterlet-addon-workmgr-bbcb9d47c-b7tcq\" (UID: \"9701a90e-fb16-49b2-b66a-dc53637da155\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq"
Sep 18 16:13:58.991185 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.991184    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/afd25d28-d917-4a77-bb0a-2003215eb405-ca\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.991627 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.991265    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9701a90e-fb16-49b2-b66a-dc53637da155-klusterlet-config\") pod \"klusterlet-addon-workmgr-bbcb9d47c-b7tcq\" (UID: \"9701a90e-fb16-49b2-b66a-dc53637da155\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq"
Sep 18 16:13:58.991627 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.991355    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afd25d28-d917-4a77-bb0a-2003215eb405-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.991627 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.991379    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9701a90e-fb16-49b2-b66a-dc53637da155-tmp\") pod \"klusterlet-addon-workmgr-bbcb9d47c-b7tcq\" (UID: \"9701a90e-fb16-49b2-b66a-dc53637da155\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq"
Sep 18 16:13:58.992059 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.992031    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd25d28-d917-4a77-bb0a-2003215eb405-additional-service-ca\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.993781 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.993761    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/afd25d28-d917-4a77-bb0a-2003215eb405-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.993870 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.993762    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/afd25d28-d917-4a77-bb0a-2003215eb405-ca\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.994230 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.994202    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9701a90e-fb16-49b2-b66a-dc53637da155-klusterlet-config\") pod \"klusterlet-addon-workmgr-bbcb9d47c-b7tcq\" (UID: \"9701a90e-fb16-49b2-b66a-dc53637da155\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq"
Sep 18 16:13:58.994453 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.994431    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/afd25d28-d917-4a77-bb0a-2003215eb405-hub\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:58.994502 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:58.994488    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afd25d28-d917-4a77-bb0a-2003215eb405-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:59.004876 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:59.004853    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctlhz\" (UniqueName: \"kubernetes.io/projected/9701a90e-fb16-49b2-b66a-dc53637da155-kube-api-access-ctlhz\") pod \"klusterlet-addon-workmgr-bbcb9d47c-b7tcq\" (UID: \"9701a90e-fb16-49b2-b66a-dc53637da155\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq"
Sep 18 16:13:59.005181 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:59.005163    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4gxk\" (UniqueName: \"kubernetes.io/projected/afd25d28-d917-4a77-bb0a-2003215eb405-kube-api-access-n4gxk\") pod \"cluster-proxy-proxy-agent-cbbc77487-b5zzr\" (UID: \"afd25d28-d917-4a77-bb0a-2003215eb405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:59.174419 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:59.174341    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq"
Sep 18 16:13:59.180606 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:59.180591    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"
Sep 18 16:13:59.454733 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:13:59.454703    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9701a90e_fb16_49b2_b66a_dc53637da155.slice/crio-ce1270c400659d275b4156568e5aacb7995c2b97826096ce3f5e5d59b0577b82 WatchSource:0}: Error finding container ce1270c400659d275b4156568e5aacb7995c2b97826096ce3f5e5d59b0577b82: Status 404 returned error can't find the container with id ce1270c400659d275b4156568e5aacb7995c2b97826096ce3f5e5d59b0577b82
Sep 18 16:13:59.460186 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:59.460165    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq"]
Sep 18 16:13:59.473279 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:13:59.473255    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafd25d28_d917_4a77_bb0a_2003215eb405.slice/crio-f82648303d655c599a25941a6db8768b6388a7d4b19eefa15eca38a4f1b47d03 WatchSource:0}: Error finding container f82648303d655c599a25941a6db8768b6388a7d4b19eefa15eca38a4f1b47d03: Status 404 returned error can't find the container with id f82648303d655c599a25941a6db8768b6388a7d4b19eefa15eca38a4f1b47d03
Sep 18 16:13:59.473996 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:59.473975    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr"]
Sep 18 16:13:59.510905 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:59.510879    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq" event={"ID":"9701a90e-fb16-49b2-b66a-dc53637da155","Type":"ContainerStarted","Data":"ce1270c400659d275b4156568e5aacb7995c2b97826096ce3f5e5d59b0577b82"}
Sep 18 16:13:59.512110 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:13:59.512088    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr" event={"ID":"afd25d28-d917-4a77-bb0a-2003215eb405","Type":"ContainerStarted","Data":"f82648303d655c599a25941a6db8768b6388a7d4b19eefa15eca38a4f1b47d03"}
Sep 18 16:14:05.548275 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:05.548234    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr" event={"ID":"afd25d28-d917-4a77-bb0a-2003215eb405","Type":"ContainerStarted","Data":"304a3a9d9f68629f21b5559d1350ff730c65d580e30e749c77054e9aad17db2c"}
Sep 18 16:14:05.548791 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:05.548285    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr" event={"ID":"afd25d28-d917-4a77-bb0a-2003215eb405","Type":"ContainerStarted","Data":"2661356c86d63cebae9abf46f21172fa7f89e52de93d589a0b332289d31b2efd"}
Sep 18 16:14:05.548791 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:05.548301    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr" event={"ID":"afd25d28-d917-4a77-bb0a-2003215eb405","Type":"ContainerStarted","Data":"39919a74519a1de45ce6a7991a362d28a870ff5c1ce63f76fe67eb29bd96b7e3"}
Sep 18 16:14:05.549680 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:05.549657    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq" event={"ID":"9701a90e-fb16-49b2-b66a-dc53637da155","Type":"ContainerStarted","Data":"caf937b7697bd4764630115c4124767ba290c1329a9903c61694feab0b46017e"}
Sep 18 16:14:05.549860 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:05.549849    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq"
Sep 18 16:14:05.551411 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:05.551389    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq"
Sep 18 16:14:05.572035 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:05.571990    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cbbc77487-b5zzr" podStartSLOduration=2.051673512 podStartE2EDuration="7.571976913s" totalImagesPullingTime="5.520303401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:14:05.570472302 +0000 UTC m=+540.517971070" watchObservedRunningTime="2026-09-18 16:14:05.571976913 +0000 UTC m=+540.519475688"
Sep 18 16:14:05.594942 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:05.594902    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bbcb9d47c-b7tcq" podStartSLOduration=2.05861955 podStartE2EDuration="7.594893562s" totalImagesPullingTime="5.536274012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:14:05.593663697 +0000 UTC m=+540.541162461" watchObservedRunningTime="2026-09-18 16:14:05.594893562 +0000 UTC m=+540.542392336"
Sep 18 16:14:16.651185 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:16.651153    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-wpxt7"]
Sep 18 16:14:16.654727 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:16.654708    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wpxt7"
Sep 18 16:14:16.658157 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:16.658130    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vd96t\""
Sep 18 16:14:16.670410 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:16.670388    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wpxt7"]
Sep 18 16:14:16.753928 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:16.753894    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b98xb\" (UniqueName: \"kubernetes.io/projected/55b61b08-66e8-466d-8a9f-d29946ae5c19-kube-api-access-b98xb\") pod \"authorino-59c974bd48-wpxt7\" (UID: \"55b61b08-66e8-466d-8a9f-d29946ae5c19\") " pod="kuadrant-system/authorino-59c974bd48-wpxt7"
Sep 18 16:14:16.855254 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:16.855216    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b98xb\" (UniqueName: \"kubernetes.io/projected/55b61b08-66e8-466d-8a9f-d29946ae5c19-kube-api-access-b98xb\") pod \"authorino-59c974bd48-wpxt7\" (UID: \"55b61b08-66e8-466d-8a9f-d29946ae5c19\") " pod="kuadrant-system/authorino-59c974bd48-wpxt7"
Sep 18 16:14:16.870948 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:16.870924    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b98xb\" (UniqueName: \"kubernetes.io/projected/55b61b08-66e8-466d-8a9f-d29946ae5c19-kube-api-access-b98xb\") pod \"authorino-59c974bd48-wpxt7\" (UID: \"55b61b08-66e8-466d-8a9f-d29946ae5c19\") " pod="kuadrant-system/authorino-59c974bd48-wpxt7"
Sep 18 16:14:16.963044 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:16.963021    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wpxt7"
Sep 18 16:14:17.240943 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:17.240870    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wpxt7"]
Sep 18 16:14:17.244466 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:14:17.244438    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55b61b08_66e8_466d_8a9f_d29946ae5c19.slice/crio-d1d2f9293a07ecc80cc56f7621471d2b3bf8c1348a04c4de38bc189ecf430637 WatchSource:0}: Error finding container d1d2f9293a07ecc80cc56f7621471d2b3bf8c1348a04c4de38bc189ecf430637: Status 404 returned error can't find the container with id d1d2f9293a07ecc80cc56f7621471d2b3bf8c1348a04c4de38bc189ecf430637
Sep 18 16:14:17.604505 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:17.604419    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wpxt7" event={"ID":"55b61b08-66e8-466d-8a9f-d29946ae5c19","Type":"ContainerStarted","Data":"d1d2f9293a07ecc80cc56f7621471d2b3bf8c1348a04c4de38bc189ecf430637"}
Sep 18 16:14:19.616907 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:19.616868    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wpxt7" event={"ID":"55b61b08-66e8-466d-8a9f-d29946ae5c19","Type":"ContainerStarted","Data":"c0d7f0f50066c38b85aea5976f1e2b44fd762b1d4411c38a8c839f8d3e1d8f91"}
Sep 18 16:14:46.065843 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:46.065784    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-wpxt7" podStartSLOduration=28.442889581 podStartE2EDuration="30.065767134s" totalImagesPullingTime="1.622877553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:14:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:14:19.6637544 +0000 UTC m=+554.611253183" watchObservedRunningTime="2026-09-18 16:14:46.065767134 +0000 UTC m=+581.013265904"
Sep 18 16:14:46.066265 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:46.066040    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wpxt7"]
Sep 18 16:14:46.066265 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:46.066236    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-wpxt7" podUID="55b61b08-66e8-466d-8a9f-d29946ae5c19" containerName="authorino" containerID="cri-o://c0d7f0f50066c38b85aea5976f1e2b44fd762b1d4411c38a8c839f8d3e1d8f91" gracePeriod=30
Sep 18 16:14:46.302137 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:46.302113    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wpxt7"
Sep 18 16:14:46.391114 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:46.391024    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b98xb\" (UniqueName: \"kubernetes.io/projected/55b61b08-66e8-466d-8a9f-d29946ae5c19-kube-api-access-b98xb\") pod \"55b61b08-66e8-466d-8a9f-d29946ae5c19\" (UID: \"55b61b08-66e8-466d-8a9f-d29946ae5c19\") "
Sep 18 16:14:46.393534 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:46.393489    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b61b08-66e8-466d-8a9f-d29946ae5c19-kube-api-access-b98xb" (OuterVolumeSpecName: "kube-api-access-b98xb") pod "55b61b08-66e8-466d-8a9f-d29946ae5c19" (UID: "55b61b08-66e8-466d-8a9f-d29946ae5c19"). InnerVolumeSpecName "kube-api-access-b98xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:14:46.492131 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:46.492104    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b98xb\" (UniqueName: \"kubernetes.io/projected/55b61b08-66e8-466d-8a9f-d29946ae5c19-kube-api-access-b98xb\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:14:46.739344 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:46.739305    2554 generic.go:356] "Generic (PLEG): container finished" podID="55b61b08-66e8-466d-8a9f-d29946ae5c19" containerID="c0d7f0f50066c38b85aea5976f1e2b44fd762b1d4411c38a8c839f8d3e1d8f91" exitCode=0
Sep 18 16:14:46.739566 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:46.739353    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wpxt7"
Sep 18 16:14:46.739566 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:46.739401    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wpxt7" event={"ID":"55b61b08-66e8-466d-8a9f-d29946ae5c19","Type":"ContainerDied","Data":"c0d7f0f50066c38b85aea5976f1e2b44fd762b1d4411c38a8c839f8d3e1d8f91"}
Sep 18 16:14:46.739566 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:46.739447    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wpxt7" event={"ID":"55b61b08-66e8-466d-8a9f-d29946ae5c19","Type":"ContainerDied","Data":"d1d2f9293a07ecc80cc56f7621471d2b3bf8c1348a04c4de38bc189ecf430637"}
Sep 18 16:14:46.739566 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:46.739471    2554 scope.go:117] "RemoveContainer" containerID="c0d7f0f50066c38b85aea5976f1e2b44fd762b1d4411c38a8c839f8d3e1d8f91"
Sep 18 16:14:46.750498 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:46.750483    2554 scope.go:117] "RemoveContainer" containerID="c0d7f0f50066c38b85aea5976f1e2b44fd762b1d4411c38a8c839f8d3e1d8f91"
Sep 18 16:14:46.750791 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:14:46.750773    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d7f0f50066c38b85aea5976f1e2b44fd762b1d4411c38a8c839f8d3e1d8f91\": container with ID starting with c0d7f0f50066c38b85aea5976f1e2b44fd762b1d4411c38a8c839f8d3e1d8f91 not found: ID does not exist" containerID="c0d7f0f50066c38b85aea5976f1e2b44fd762b1d4411c38a8c839f8d3e1d8f91"
Sep 18 16:14:46.750857 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:46.750806    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d7f0f50066c38b85aea5976f1e2b44fd762b1d4411c38a8c839f8d3e1d8f91"} err="failed to get container status \"c0d7f0f50066c38b85aea5976f1e2b44fd762b1d4411c38a8c839f8d3e1d8f91\": rpc error: code = NotFound desc = could not find container \"c0d7f0f50066c38b85aea5976f1e2b44fd762b1d4411c38a8c839f8d3e1d8f91\": container with ID starting with c0d7f0f50066c38b85aea5976f1e2b44fd762b1d4411c38a8c839f8d3e1d8f91 not found: ID does not exist"
Sep 18 16:14:46.799176 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:46.799145    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wpxt7"]
Sep 18 16:14:46.806139 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:46.806117    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wpxt7"]
Sep 18 16:14:47.278795 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:14:47.278757    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55b61b08-66e8-466d-8a9f-d29946ae5c19" path="/var/lib/kubelet/pods/55b61b08-66e8-466d-8a9f-d29946ae5c19/volumes"
Sep 18 16:15:55.999203 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:55.999161    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"]
Sep 18 16:15:56.001649 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:55.999690    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55b61b08-66e8-466d-8a9f-d29946ae5c19" containerName="authorino"
Sep 18 16:15:56.001649 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:55.999715    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="55b61b08-66e8-466d-8a9f-d29946ae5c19" containerName="authorino"
Sep 18 16:15:56.001649 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:55.999788    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="55b61b08-66e8-466d-8a9f-d29946ae5c19" containerName="authorino"
Sep 18 16:15:56.002760 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.002723    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.006922 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.006897    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 16:15:56.007033 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.006897    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 16:15:56.007748 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.007730    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 18 16:15:56.007850 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.007793    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sr7qq\""
Sep 18 16:15:56.027468 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.027445    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"]
Sep 18 16:15:56.060144 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.060114    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4da75786-59ac-4f5e-843f-cb4eba985284-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.060249 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.060147    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da75786-59ac-4f5e-843f-cb4eba985284-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.060249 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.060229    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6847c\" (UniqueName: \"kubernetes.io/projected/4da75786-59ac-4f5e-843f-cb4eba985284-kube-api-access-6847c\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.060329 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.060316    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4da75786-59ac-4f5e-843f-cb4eba985284-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.060367 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.060340    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4da75786-59ac-4f5e-843f-cb4eba985284-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.060412 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.060379    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4da75786-59ac-4f5e-843f-cb4eba985284-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.161018 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.160979    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4da75786-59ac-4f5e-843f-cb4eba985284-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.161018 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.161024    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da75786-59ac-4f5e-843f-cb4eba985284-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.161217 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.161072    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6847c\" (UniqueName: \"kubernetes.io/projected/4da75786-59ac-4f5e-843f-cb4eba985284-kube-api-access-6847c\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.161217 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.161129    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4da75786-59ac-4f5e-843f-cb4eba985284-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.161328 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.161314    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4da75786-59ac-4f5e-843f-cb4eba985284-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.161381 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.161355    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4da75786-59ac-4f5e-843f-cb4eba985284-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.161434 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.161403    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4da75786-59ac-4f5e-843f-cb4eba985284-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.161434 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.161418    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4da75786-59ac-4f5e-843f-cb4eba985284-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.161679 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.161660    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4da75786-59ac-4f5e-843f-cb4eba985284-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.163566 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.163546    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4da75786-59ac-4f5e-843f-cb4eba985284-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.163714 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.163688    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4da75786-59ac-4f5e-843f-cb4eba985284-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.176753 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.176735    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6847c\" (UniqueName: \"kubernetes.io/projected/4da75786-59ac-4f5e-843f-cb4eba985284-kube-api-access-6847c\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd\" (UID: \"4da75786-59ac-4f5e-843f-cb4eba985284\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.315008 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.314942    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:15:56.602289 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:56.602205    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"]
Sep 18 16:15:56.604494 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:15:56.604463    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da75786_59ac_4f5e_843f_cb4eba985284.slice/crio-abfcbc7dde16ae8739723d1395d2b611bdade7044cfa3327d179087476e46dc6 WatchSource:0}: Error finding container abfcbc7dde16ae8739723d1395d2b611bdade7044cfa3327d179087476e46dc6: Status 404 returned error can't find the container with id abfcbc7dde16ae8739723d1395d2b611bdade7044cfa3327d179087476e46dc6
Sep 18 16:15:57.051120 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:57.051075    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd" event={"ID":"4da75786-59ac-4f5e-843f-cb4eba985284","Type":"ContainerStarted","Data":"abfcbc7dde16ae8739723d1395d2b611bdade7044cfa3327d179087476e46dc6"}
Sep 18 16:15:58.057559 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:58.057508    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd" event={"ID":"4da75786-59ac-4f5e-843f-cb4eba985284","Type":"ContainerStarted","Data":"4514a8b2a68219225c0a94521e94498f3c693f2ebdafdc03428d7170f7dda44c"}
Sep 18 16:15:58.123714 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:15:58.123618    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd" podStartSLOduration=1.900153515 podStartE2EDuration="3.123602911s" totalImagesPullingTime="1.223449396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:58.11334921 +0000 UTC m=+653.060847984" watchObservedRunningTime="2026-09-18 16:15:58.123602911 +0000 UTC m=+653.071101684"
Sep 18 16:16:06.315605 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:06.315551    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:16:06.316087 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:06.315638    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:16:06.328285 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:06.328256    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:16:07.110043 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:07.110014    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd"
Sep 18 16:16:27.085955 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.085924    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"]
Sep 18 16:16:27.089152 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.089136    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.106938 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.106916    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 18 16:16:27.117979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.117955    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"]
Sep 18 16:16:27.218635 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.218605    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb38610a-7d3c-4a60-8933-0cbe967b2d36-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.218767 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.218646    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh8gx\" (UniqueName: \"kubernetes.io/projected/fb38610a-7d3c-4a60-8933-0cbe967b2d36-kube-api-access-kh8gx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.218767 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.218687    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb38610a-7d3c-4a60-8933-0cbe967b2d36-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.218767 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.218727    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb38610a-7d3c-4a60-8933-0cbe967b2d36-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.218767 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.218752    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb38610a-7d3c-4a60-8933-0cbe967b2d36-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.218898 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.218780    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb38610a-7d3c-4a60-8933-0cbe967b2d36-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.319744 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.319700    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb38610a-7d3c-4a60-8933-0cbe967b2d36-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.319866 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.319762    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb38610a-7d3c-4a60-8933-0cbe967b2d36-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.319866 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.319787    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh8gx\" (UniqueName: \"kubernetes.io/projected/fb38610a-7d3c-4a60-8933-0cbe967b2d36-kube-api-access-kh8gx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.319866 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.319806    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb38610a-7d3c-4a60-8933-0cbe967b2d36-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.319866 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.319835    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb38610a-7d3c-4a60-8933-0cbe967b2d36-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.319866 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.319854    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb38610a-7d3c-4a60-8933-0cbe967b2d36-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.320147 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.320127    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb38610a-7d3c-4a60-8933-0cbe967b2d36-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.320193 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.320141    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb38610a-7d3c-4a60-8933-0cbe967b2d36-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.320193 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.320184    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb38610a-7d3c-4a60-8933-0cbe967b2d36-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.322234 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.322211    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb38610a-7d3c-4a60-8933-0cbe967b2d36-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.322462 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.322445    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb38610a-7d3c-4a60-8933-0cbe967b2d36-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.330147 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.330124    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh8gx\" (UniqueName: \"kubernetes.io/projected/fb38610a-7d3c-4a60-8933-0cbe967b2d36-kube-api-access-kh8gx\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-khw86\" (UID: \"fb38610a-7d3c-4a60-8933-0cbe967b2d36\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.399165 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.399102    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:27.686561 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.686512    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"]
Sep 18 16:16:27.690349 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:16:27.690315    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb38610a_7d3c_4a60_8933_0cbe967b2d36.slice/crio-39c3d64a0a693718e888e50780619e93c56fad65ec3379ecc5cbc13f3adca0d7 WatchSource:0}: Error finding container 39c3d64a0a693718e888e50780619e93c56fad65ec3379ecc5cbc13f3adca0d7: Status 404 returned error can't find the container with id 39c3d64a0a693718e888e50780619e93c56fad65ec3379ecc5cbc13f3adca0d7
Sep 18 16:16:27.691789 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:27.691768    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:16:28.194091 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:28.194015    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86" event={"ID":"fb38610a-7d3c-4a60-8933-0cbe967b2d36","Type":"ContainerStarted","Data":"1d283b3466ec8644ec21fd894eb241bf878a35a29ec167d29300f59edf7da0cd"}
Sep 18 16:16:28.194091 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:28.194072    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86" event={"ID":"fb38610a-7d3c-4a60-8933-0cbe967b2d36","Type":"ContainerStarted","Data":"39c3d64a0a693718e888e50780619e93c56fad65ec3379ecc5cbc13f3adca0d7"}
Sep 18 16:16:28.234988 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:28.234941    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86" podStartSLOduration=1.037138072 podStartE2EDuration="1.234916951s" totalImagesPullingTime="197.778879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:28.234331301 +0000 UTC m=+683.181830074" watchObservedRunningTime="2026-09-18 16:16:28.234916951 +0000 UTC m=+683.182415724"
Sep 18 16:16:37.399274 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:37.399242    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:37.399859 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:37.399326    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:37.412103 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:37.412083    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:38.249228 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:38.249204    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-khw86"
Sep 18 16:16:58.954684 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:58.954612    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"]
Sep 18 16:16:58.958086 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:58.958068    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:58.986667 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:58.986642    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 18 16:16:59.001835 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.001808    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"]
Sep 18 16:16:59.087384 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.087351    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/313dd42f-6969-42f5-8fdf-a1a119aa710c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.087384 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.087388    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/313dd42f-6969-42f5-8fdf-a1a119aa710c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.087615 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.087427    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8r9f\" (UniqueName: \"kubernetes.io/projected/313dd42f-6969-42f5-8fdf-a1a119aa710c-kube-api-access-s8r9f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.087615 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.087446    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/313dd42f-6969-42f5-8fdf-a1a119aa710c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.087615 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.087467    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/313dd42f-6969-42f5-8fdf-a1a119aa710c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.087615 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.087557    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/313dd42f-6969-42f5-8fdf-a1a119aa710c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.189000 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.188973    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/313dd42f-6969-42f5-8fdf-a1a119aa710c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.189000 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.189003    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/313dd42f-6969-42f5-8fdf-a1a119aa710c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.189241 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.189055    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8r9f\" (UniqueName: \"kubernetes.io/projected/313dd42f-6969-42f5-8fdf-a1a119aa710c-kube-api-access-s8r9f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.189241 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.189085    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/313dd42f-6969-42f5-8fdf-a1a119aa710c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.189241 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.189115    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/313dd42f-6969-42f5-8fdf-a1a119aa710c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.189241 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.189154    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/313dd42f-6969-42f5-8fdf-a1a119aa710c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.189440 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.189401    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/313dd42f-6969-42f5-8fdf-a1a119aa710c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.189484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.189447    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/313dd42f-6969-42f5-8fdf-a1a119aa710c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.189558 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.189516    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/313dd42f-6969-42f5-8fdf-a1a119aa710c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.191538 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.191499    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/313dd42f-6969-42f5-8fdf-a1a119aa710c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.191791 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.191774    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/313dd42f-6969-42f5-8fdf-a1a119aa710c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.199740 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.199720    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8r9f\" (UniqueName: \"kubernetes.io/projected/313dd42f-6969-42f5-8fdf-a1a119aa710c-kube-api-access-s8r9f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk\" (UID: \"313dd42f-6969-42f5-8fdf-a1a119aa710c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.268742 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.268669    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:16:59.553389 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:16:59.553311    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"]
Sep 18 16:16:59.559445 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:16:59.559415    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313dd42f_6969_42f5_8fdf_a1a119aa710c.slice/crio-a47bf3e0a3bf82dc131c210f59cade94cbf283700896c1aa6f617080235e43c8 WatchSource:0}: Error finding container a47bf3e0a3bf82dc131c210f59cade94cbf283700896c1aa6f617080235e43c8: Status 404 returned error can't find the container with id a47bf3e0a3bf82dc131c210f59cade94cbf283700896c1aa6f617080235e43c8
Sep 18 16:17:00.338350 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:00.338313    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk" event={"ID":"313dd42f-6969-42f5-8fdf-a1a119aa710c","Type":"ContainerStarted","Data":"647e81cbdab14299b6739bae8335f25c6ed45bfaf6a1b50359bf18011c13bfed"}
Sep 18 16:17:00.338756 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:00.338373    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk" event={"ID":"313dd42f-6969-42f5-8fdf-a1a119aa710c","Type":"ContainerStarted","Data":"a47bf3e0a3bf82dc131c210f59cade94cbf283700896c1aa6f617080235e43c8"}
Sep 18 16:17:00.373408 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:00.373351    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk" podStartSLOduration=2.2077405 podStartE2EDuration="2.37333406s" totalImagesPullingTime="165.59356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:00.372921728 +0000 UTC m=+715.320420503" watchObservedRunningTime="2026-09-18 16:17:00.37333406 +0000 UTC m=+715.320832833"
Sep 18 16:17:06.367368 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.367327    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"]
Sep 18 16:17:06.370936 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.370919    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.374296 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.374277    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 18 16:17:06.385696 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.385673    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"]
Sep 18 16:17:06.460970 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.460937    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngkqg\" (UniqueName: \"kubernetes.io/projected/f477ec87-5a46-4d37-8010-804723ca9841-kube-api-access-ngkqg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.461111 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.461005    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f477ec87-5a46-4d37-8010-804723ca9841-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.461111 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.461074    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f477ec87-5a46-4d37-8010-804723ca9841-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.461111 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.461106    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f477ec87-5a46-4d37-8010-804723ca9841-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.461214 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.461158    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f477ec87-5a46-4d37-8010-804723ca9841-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.461249 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.461237    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f477ec87-5a46-4d37-8010-804723ca9841-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.562015 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.561979    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkqg\" (UniqueName: \"kubernetes.io/projected/f477ec87-5a46-4d37-8010-804723ca9841-kube-api-access-ngkqg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.562161 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.562029    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f477ec87-5a46-4d37-8010-804723ca9841-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.562161 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.562054    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f477ec87-5a46-4d37-8010-804723ca9841-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.562252 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.562173    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f477ec87-5a46-4d37-8010-804723ca9841-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.562323 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.562259    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f477ec87-5a46-4d37-8010-804723ca9841-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.562323 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.562315    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f477ec87-5a46-4d37-8010-804723ca9841-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.562454 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.562437    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f477ec87-5a46-4d37-8010-804723ca9841-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.562556 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.562517    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f477ec87-5a46-4d37-8010-804723ca9841-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.562696 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.562631    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f477ec87-5a46-4d37-8010-804723ca9841-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.564253 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.564236    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f477ec87-5a46-4d37-8010-804723ca9841-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.564626 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.564610    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f477ec87-5a46-4d37-8010-804723ca9841-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.577513 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.577492    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngkqg\" (UniqueName: \"kubernetes.io/projected/f477ec87-5a46-4d37-8010-804723ca9841-kube-api-access-ngkqg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mwprb\" (UID: \"f477ec87-5a46-4d37-8010-804723ca9841\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.681146 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.681122    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:06.988659 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:06.988575    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"]
Sep 18 16:17:06.996138 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:17:06.996111    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf477ec87_5a46_4d37_8010_804723ca9841.slice/crio-a9dfdbb43d2f8c75d76b9cc86ca8a0083220b4c21dad71d8131d6a339be7caa7 WatchSource:0}: Error finding container a9dfdbb43d2f8c75d76b9cc86ca8a0083220b4c21dad71d8131d6a339be7caa7: Status 404 returned error can't find the container with id a9dfdbb43d2f8c75d76b9cc86ca8a0083220b4c21dad71d8131d6a339be7caa7
Sep 18 16:17:07.378583 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:07.378480    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb" event={"ID":"f477ec87-5a46-4d37-8010-804723ca9841","Type":"ContainerStarted","Data":"c21fccd78d11e30a473f82fdf2e0f130febd8af592df7c8e6ed986af073b6b51"}
Sep 18 16:17:07.378949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:07.378581    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb" event={"ID":"f477ec87-5a46-4d37-8010-804723ca9841","Type":"ContainerStarted","Data":"a9dfdbb43d2f8c75d76b9cc86ca8a0083220b4c21dad71d8131d6a339be7caa7"}
Sep 18 16:17:07.407823 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:07.407769    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb" podStartSLOduration=1.222281694 podStartE2EDuration="1.407750206s" totalImagesPullingTime="185.468512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:07.406411422 +0000 UTC m=+722.353910197" watchObservedRunningTime="2026-09-18 16:17:07.407750206 +0000 UTC m=+722.355248983"
Sep 18 16:17:09.269090 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:09.269057    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:17:09.269457 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:09.269116    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:17:09.282168 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:09.282139    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:17:09.399825 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:09.399794    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk"
Sep 18 16:17:16.681625 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:16.681589    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:16.682020 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:16.681648    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:16.694138 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:16.694115    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:17:17.436766 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:17:17.436735    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mwprb"
Sep 18 16:19:30.210161 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.210126    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"]
Sep 18 16:19:30.213718 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.213701    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:30.217147 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.217120    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:19:30.217271 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.217121    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fbc529-serving-cert\""
Sep 18 16:19:30.218182 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.218160    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n9smt\""
Sep 18 16:19:30.218182 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.218189    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 16:19:30.219341 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.219323    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 16:19:30.240177 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.240154    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"]
Sep 18 16:19:30.325173 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.325137    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380b3a73-8056-4f47-9005-00e48e7a0b32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:30.325317 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.325183    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msw5t\" (UniqueName: \"kubernetes.io/projected/380b3a73-8056-4f47-9005-00e48e7a0b32-kube-api-access-msw5t\") pod \"maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:30.325317 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.325255    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380b3a73-8056-4f47-9005-00e48e7a0b32-metrics-certs\") pod \"maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:30.325411 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.325318    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380b3a73-8056-4f47-9005-00e48e7a0b32-maas-api-tls\") pod \"maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:30.425963 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.425924    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380b3a73-8056-4f47-9005-00e48e7a0b32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:30.425963 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.425967    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msw5t\" (UniqueName: \"kubernetes.io/projected/380b3a73-8056-4f47-9005-00e48e7a0b32-kube-api-access-msw5t\") pod \"maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:30.426208 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.426004    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380b3a73-8056-4f47-9005-00e48e7a0b32-metrics-certs\") pod \"maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:30.426208 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.426067    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380b3a73-8056-4f47-9005-00e48e7a0b32-maas-api-tls\") pod \"maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:30.426208 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:19:30.426169    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-serving-cert: secret "maas-api-e2e-inf-a-fbc529-serving-cert" not found
Sep 18 16:19:30.426360 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:19:30.426237    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/380b3a73-8056-4f47-9005-00e48e7a0b32-maas-api-tls podName:380b3a73-8056-4f47-9005-00e48e7a0b32 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:30.926217438 +0000 UTC m=+865.873716195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/380b3a73-8056-4f47-9005-00e48e7a0b32-maas-api-tls") pod "maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw" (UID: "380b3a73-8056-4f47-9005-00e48e7a0b32") : secret "maas-api-e2e-inf-a-fbc529-serving-cert" not found
Sep 18 16:19:30.426747 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.426725    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380b3a73-8056-4f47-9005-00e48e7a0b32-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:30.428574 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.428557    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380b3a73-8056-4f47-9005-00e48e7a0b32-metrics-certs\") pod \"maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:30.447920 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.447895    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msw5t\" (UniqueName: \"kubernetes.io/projected/380b3a73-8056-4f47-9005-00e48e7a0b32-kube-api-access-msw5t\") pod \"maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:30.930937 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.930899    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380b3a73-8056-4f47-9005-00e48e7a0b32-maas-api-tls\") pod \"maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:30.939115 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:30.939076    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380b3a73-8056-4f47-9005-00e48e7a0b32-maas-api-tls\") pod \"maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:31.124408 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:31.124378    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:31.424655 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:31.424630    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"]
Sep 18 16:19:31.427877 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:19:31.427849    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380b3a73_8056_4f47_9005_00e48e7a0b32.slice/crio-86baeae8149a8b25ad6ab7b6d51c27065bf87bda457dcd1492d5ee2c3f422aa6 WatchSource:0}: Error finding container 86baeae8149a8b25ad6ab7b6d51c27065bf87bda457dcd1492d5ee2c3f422aa6: Status 404 returned error can't find the container with id 86baeae8149a8b25ad6ab7b6d51c27065bf87bda457dcd1492d5ee2c3f422aa6
Sep 18 16:19:32.048879 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.048837    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw" event={"ID":"380b3a73-8056-4f47-9005-00e48e7a0b32","Type":"ContainerStarted","Data":"86baeae8149a8b25ad6ab7b6d51c27065bf87bda457dcd1492d5ee2c3f422aa6"}
Sep 18 16:19:32.154577 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.154538    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"]
Sep 18 16:19:32.158576 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.158552    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:32.162153 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.161959    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-1eec1d-serving-cert\""
Sep 18 16:19:32.172033 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.170747    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"]
Sep 18 16:19:32.243292 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.243071    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbebcbf4-fa27-4811-9cad-5ab137b70841-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:32.243292 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.243124    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbebcbf4-fa27-4811-9cad-5ab137b70841-metrics-certs\") pod \"maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:32.243292 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.243190    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbebcbf4-fa27-4811-9cad-5ab137b70841-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:32.243292 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.243214    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc8bd\" (UniqueName: \"kubernetes.io/projected/fbebcbf4-fa27-4811-9cad-5ab137b70841-kube-api-access-mc8bd\") pod \"maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:32.343916 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.343833    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbebcbf4-fa27-4811-9cad-5ab137b70841-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:32.343916 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.343872    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc8bd\" (UniqueName: \"kubernetes.io/projected/fbebcbf4-fa27-4811-9cad-5ab137b70841-kube-api-access-mc8bd\") pod \"maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:32.344120 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.343916    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbebcbf4-fa27-4811-9cad-5ab137b70841-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:32.344120 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.343975    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbebcbf4-fa27-4811-9cad-5ab137b70841-metrics-certs\") pod \"maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:32.344321 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:19:32.344293    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-serving-cert: secret "maas-api-e2e-worker-w0-1eec1d-serving-cert" not found
Sep 18 16:19:32.344402 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:19:32.344374    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbebcbf4-fa27-4811-9cad-5ab137b70841-maas-api-tls podName:fbebcbf4-fa27-4811-9cad-5ab137b70841 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:32.844353975 +0000 UTC m=+867.791852736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbebcbf4-fa27-4811-9cad-5ab137b70841-maas-api-tls") pod "maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56" (UID: "fbebcbf4-fa27-4811-9cad-5ab137b70841") : secret "maas-api-e2e-worker-w0-1eec1d-serving-cert" not found
Sep 18 16:19:32.344676 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.344631    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbebcbf4-fa27-4811-9cad-5ab137b70841-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:32.347138 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.347111    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbebcbf4-fa27-4811-9cad-5ab137b70841-metrics-certs\") pod \"maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:32.359858 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.359827    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc8bd\" (UniqueName: \"kubernetes.io/projected/fbebcbf4-fa27-4811-9cad-5ab137b70841-kube-api-access-mc8bd\") pod \"maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:32.849052 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.849001    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbebcbf4-fa27-4811-9cad-5ab137b70841-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:32.851859 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:32.851805    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbebcbf4-fa27-4811-9cad-5ab137b70841-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:33.077623 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:33.077588    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:33.825352 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:33.825327    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"]
Sep 18 16:19:33.827599 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:19:33.827570    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbebcbf4_fa27_4811_9cad_5ab137b70841.slice/crio-8c7a44a6a7862d0ddada5d3c3e27e78165c94ba88efec2329e8a7737978ac298 WatchSource:0}: Error finding container 8c7a44a6a7862d0ddada5d3c3e27e78165c94ba88efec2329e8a7737978ac298: Status 404 returned error can't find the container with id 8c7a44a6a7862d0ddada5d3c3e27e78165c94ba88efec2329e8a7737978ac298
Sep 18 16:19:34.064320 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:34.064284    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw" event={"ID":"380b3a73-8056-4f47-9005-00e48e7a0b32","Type":"ContainerStarted","Data":"83609f04ddfa1dc1c8b996c02344f5b980e20fee90e5016ee5bbdf447c501654"}
Sep 18 16:19:34.064792 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:34.064426    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:34.065490 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:34.065466    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56" event={"ID":"fbebcbf4-fa27-4811-9cad-5ab137b70841","Type":"ContainerStarted","Data":"8c7a44a6a7862d0ddada5d3c3e27e78165c94ba88efec2329e8a7737978ac298"}
Sep 18 16:19:34.086019 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:34.085950    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw" podStartSLOduration=1.90451695 podStartE2EDuration="4.085927366s" totalImagesPullingTime="2.181410416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:34.084508335 +0000 UTC m=+869.032007109" watchObservedRunningTime="2026-09-18 16:19:34.085927366 +0000 UTC m=+869.033426139"
Sep 18 16:19:35.072449 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:35.072412    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56" event={"ID":"fbebcbf4-fa27-4811-9cad-5ab137b70841","Type":"ContainerStarted","Data":"ffc7a7a62fb1f68afd277de6d7b11bd82835751d4ffc4b94bb3f91726739afee"}
Sep 18 16:19:35.072888 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:35.072475    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:35.109941 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:35.109893    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56" podStartSLOduration=2.717702146 podStartE2EDuration="3.109878438s" totalImagesPullingTime="392.176292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:35.107825229 +0000 UTC m=+870.055324005" watchObservedRunningTime="2026-09-18 16:19:35.109878438 +0000 UTC m=+870.057377213"
Sep 18 16:19:37.372092 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:37.372059    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"]
Sep 18 16:19:37.375679 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:37.375662    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:37.383540 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:37.383505    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-eef18d-serving-cert\""
Sep 18 16:19:37.404349 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:37.404325    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"]
Sep 18 16:19:37.492317 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:37.492281    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-maas-api-tls\") pod \"maas-api-e2e-autores-eef18d-79b84c4558-rbg2l\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:37.492317 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:37.492320    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/474d3de6-0135-47cb-880b-3fae81cc586a-openshift-service-ca\") pod \"maas-api-e2e-autores-eef18d-79b84c4558-rbg2l\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:37.492598 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:37.492372    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4bwc\" (UniqueName: \"kubernetes.io/projected/474d3de6-0135-47cb-880b-3fae81cc586a-kube-api-access-k4bwc\") pod \"maas-api-e2e-autores-eef18d-79b84c4558-rbg2l\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:37.492598 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:37.492457    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-metrics-certs\") pod \"maas-api-e2e-autores-eef18d-79b84c4558-rbg2l\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:37.593529 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:37.593499    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-maas-api-tls\") pod \"maas-api-e2e-autores-eef18d-79b84c4558-rbg2l\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:37.593663 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:37.593560    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/474d3de6-0135-47cb-880b-3fae81cc586a-openshift-service-ca\") pod \"maas-api-e2e-autores-eef18d-79b84c4558-rbg2l\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:37.593663 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:37.593593    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4bwc\" (UniqueName: \"kubernetes.io/projected/474d3de6-0135-47cb-880b-3fae81cc586a-kube-api-access-k4bwc\") pod \"maas-api-e2e-autores-eef18d-79b84c4558-rbg2l\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:37.593663 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:37.593618    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-metrics-certs\") pod \"maas-api-e2e-autores-eef18d-79b84c4558-rbg2l\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:37.593663 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:19:37.593644    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-serving-cert: secret "maas-api-e2e-autores-eef18d-serving-cert" not found
Sep 18 16:19:37.593824 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:19:37.593703    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-maas-api-tls podName:474d3de6-0135-47cb-880b-3fae81cc586a nodeName:}" failed. No retries permitted until 2026-09-18 16:19:38.093685688 +0000 UTC m=+873.041184459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-maas-api-tls") pod "maas-api-e2e-autores-eef18d-79b84c4558-rbg2l" (UID: "474d3de6-0135-47cb-880b-3fae81cc586a") : secret "maas-api-e2e-autores-eef18d-serving-cert" not found
Sep 18 16:19:37.594252 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:37.594226    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/474d3de6-0135-47cb-880b-3fae81cc586a-openshift-service-ca\") pod \"maas-api-e2e-autores-eef18d-79b84c4558-rbg2l\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:37.596178 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:37.596153    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-metrics-certs\") pod \"maas-api-e2e-autores-eef18d-79b84c4558-rbg2l\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:37.603027 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:37.603004    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4bwc\" (UniqueName: \"kubernetes.io/projected/474d3de6-0135-47cb-880b-3fae81cc586a-kube-api-access-k4bwc\") pod \"maas-api-e2e-autores-eef18d-79b84c4558-rbg2l\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:38.098978 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:38.098942    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-maas-api-tls\") pod \"maas-api-e2e-autores-eef18d-79b84c4558-rbg2l\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:38.099169 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:19:38.099106    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-serving-cert: secret "maas-api-e2e-autores-eef18d-serving-cert" not found
Sep 18 16:19:38.099240 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:19:38.099179    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-maas-api-tls podName:474d3de6-0135-47cb-880b-3fae81cc586a nodeName:}" failed. No retries permitted until 2026-09-18 16:19:39.099158165 +0000 UTC m=+874.046656918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-maas-api-tls") pod "maas-api-e2e-autores-eef18d-79b84c4558-rbg2l" (UID: "474d3de6-0135-47cb-880b-3fae81cc586a") : secret "maas-api-e2e-autores-eef18d-serving-cert" not found
Sep 18 16:19:38.402454 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:38.402379    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"]
Sep 18 16:19:38.405979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:38.405957    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"
Sep 18 16:19:38.409036 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:38.409016    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-eef18d-dockercfg-lk7rp\""
Sep 18 16:19:38.409389 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:38.409371    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-eef18d\""
Sep 18 16:19:38.419267 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:38.419247    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"]
Sep 18 16:19:38.502180 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:38.502148    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hpxw\" (UniqueName: \"kubernetes.io/projected/622a0d89-0bce-4ffa-a580-60e532430592-kube-api-access-4hpxw\") pod \"payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8\" (UID: \"622a0d89-0bce-4ffa-a580-60e532430592\") " pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"
Sep 18 16:19:38.502353 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:38.502226    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/622a0d89-0bce-4ffa-a580-60e532430592-plugins-config-volume\") pod \"payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8\" (UID: \"622a0d89-0bce-4ffa-a580-60e532430592\") " pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"
Sep 18 16:19:38.603421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:38.603391    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hpxw\" (UniqueName: \"kubernetes.io/projected/622a0d89-0bce-4ffa-a580-60e532430592-kube-api-access-4hpxw\") pod \"payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8\" (UID: \"622a0d89-0bce-4ffa-a580-60e532430592\") " pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"
Sep 18 16:19:38.603604 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:38.603447    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/622a0d89-0bce-4ffa-a580-60e532430592-plugins-config-volume\") pod \"payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8\" (UID: \"622a0d89-0bce-4ffa-a580-60e532430592\") " pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"
Sep 18 16:19:38.604144 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:38.604125    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/622a0d89-0bce-4ffa-a580-60e532430592-plugins-config-volume\") pod \"payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8\" (UID: \"622a0d89-0bce-4ffa-a580-60e532430592\") " pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"
Sep 18 16:19:38.613390 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:38.613368    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hpxw\" (UniqueName: \"kubernetes.io/projected/622a0d89-0bce-4ffa-a580-60e532430592-kube-api-access-4hpxw\") pod \"payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8\" (UID: \"622a0d89-0bce-4ffa-a580-60e532430592\") " pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"
Sep 18 16:19:38.716396 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:38.716372    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"
Sep 18 16:19:39.002404 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:39.002370    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"]
Sep 18 16:19:39.002558 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:19:39.002460    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622a0d89_0bce_4ffa_a580_60e532430592.slice/crio-229ef154a46cdbd6df300703e79d854075cf205bbd1016beef7a8e1a9367e9b7 WatchSource:0}: Error finding container 229ef154a46cdbd6df300703e79d854075cf205bbd1016beef7a8e1a9367e9b7: Status 404 returned error can't find the container with id 229ef154a46cdbd6df300703e79d854075cf205bbd1016beef7a8e1a9367e9b7
Sep 18 16:19:39.092653 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:39.092616    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8" event={"ID":"622a0d89-0bce-4ffa-a580-60e532430592","Type":"ContainerStarted","Data":"229ef154a46cdbd6df300703e79d854075cf205bbd1016beef7a8e1a9367e9b7"}
Sep 18 16:19:39.106919 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:39.106888    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-maas-api-tls\") pod \"maas-api-e2e-autores-eef18d-79b84c4558-rbg2l\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:39.107039 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:19:39.107018    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-serving-cert: secret "maas-api-e2e-autores-eef18d-serving-cert" not found
Sep 18 16:19:39.107104 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:19:39.107083    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-maas-api-tls podName:474d3de6-0135-47cb-880b-3fae81cc586a nodeName:}" failed. No retries permitted until 2026-09-18 16:19:41.107064049 +0000 UTC m=+876.054562824 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-maas-api-tls") pod "maas-api-e2e-autores-eef18d-79b84c4558-rbg2l" (UID: "474d3de6-0135-47cb-880b-3fae81cc586a") : secret "maas-api-e2e-autores-eef18d-serving-cert" not found
Sep 18 16:19:40.079395 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:40.079362    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:19:41.084126 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:41.084088    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:19:41.128222 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:41.128193    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-maas-api-tls\") pod \"maas-api-e2e-autores-eef18d-79b84c4558-rbg2l\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:41.131612 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:41.131515    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-maas-api-tls\") pod \"maas-api-e2e-autores-eef18d-79b84c4558-rbg2l\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:41.285405 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:41.285377    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:41.842548 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:41.842507    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"]
Sep 18 16:19:41.844208 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:19:41.844176    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474d3de6_0135_47cb_880b_3fae81cc586a.slice/crio-63bc9b47ddabb0eb5029eb1643da213977803559c6c2f988ed3d623ab4b1e8f0 WatchSource:0}: Error finding container 63bc9b47ddabb0eb5029eb1643da213977803559c6c2f988ed3d623ab4b1e8f0: Status 404 returned error can't find the container with id 63bc9b47ddabb0eb5029eb1643da213977803559c6c2f988ed3d623ab4b1e8f0
Sep 18 16:19:42.111165 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.111088    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l" event={"ID":"474d3de6-0135-47cb-880b-3fae81cc586a","Type":"ContainerStarted","Data":"63bc9b47ddabb0eb5029eb1643da213977803559c6c2f988ed3d623ab4b1e8f0"}
Sep 18 16:19:42.112731 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.112708    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8" event={"ID":"622a0d89-0bce-4ffa-a580-60e532430592","Type":"ContainerStarted","Data":"270f50a122c7ad4441199b7b3b6ab360c1d3bf00e6bd7673b759aa9a8784a7b5"}
Sep 18 16:19:42.112844 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.112814    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"
Sep 18 16:19:42.140342 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.140299    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8" podStartSLOduration=1.5108161199999999 podStartE2EDuration="4.14028304s" totalImagesPullingTime="2.62946692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:42.138216615 +0000 UTC m=+877.085715391" watchObservedRunningTime="2026-09-18 16:19:42.14028304 +0000 UTC m=+877.087781818"
Sep 18 16:19:42.357611 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.357513    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"]
Sep 18 16:19:42.361158 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.361142    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:42.365858 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.365818    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-75688a34-serving-cert\""
Sep 18 16:19:42.378866 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.378842    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"]
Sep 18 16:19:42.541993 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.541956    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4brs\" (UniqueName: \"kubernetes.io/projected/91886522-a6e7-46e7-b2ad-6936049b4eef-kube-api-access-m4brs\") pod \"maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:42.542140 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.542009    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91886522-a6e7-46e7-b2ad-6936049b4eef-maas-api-tls\") pod \"maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:42.542140 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.542078    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91886522-a6e7-46e7-b2ad-6936049b4eef-openshift-service-ca\") pod \"maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:42.542218 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.542146    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91886522-a6e7-46e7-b2ad-6936049b4eef-metrics-certs\") pod \"maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:42.643166 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.643088    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4brs\" (UniqueName: \"kubernetes.io/projected/91886522-a6e7-46e7-b2ad-6936049b4eef-kube-api-access-m4brs\") pod \"maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:42.643166 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.643128    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91886522-a6e7-46e7-b2ad-6936049b4eef-maas-api-tls\") pod \"maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:42.643166 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.643152    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91886522-a6e7-46e7-b2ad-6936049b4eef-openshift-service-ca\") pod \"maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:42.643366 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.643211    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91886522-a6e7-46e7-b2ad-6936049b4eef-metrics-certs\") pod \"maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:42.643405 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:19:42.643368    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-serving-cert: secret "maas-api-e2e-ait-75688a34-serving-cert" not found
Sep 18 16:19:42.643452 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:19:42.643437    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91886522-a6e7-46e7-b2ad-6936049b4eef-maas-api-tls podName:91886522-a6e7-46e7-b2ad-6936049b4eef nodeName:}" failed. No retries permitted until 2026-09-18 16:19:43.143415366 +0000 UTC m=+878.090914134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91886522-a6e7-46e7-b2ad-6936049b4eef-maas-api-tls") pod "maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp" (UID: "91886522-a6e7-46e7-b2ad-6936049b4eef") : secret "maas-api-e2e-ait-75688a34-serving-cert" not found
Sep 18 16:19:42.643831 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.643814    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91886522-a6e7-46e7-b2ad-6936049b4eef-openshift-service-ca\") pod \"maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:42.645952 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.645931    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91886522-a6e7-46e7-b2ad-6936049b4eef-metrics-certs\") pod \"maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:42.658907 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:42.658887    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4brs\" (UniqueName: \"kubernetes.io/projected/91886522-a6e7-46e7-b2ad-6936049b4eef-kube-api-access-m4brs\") pod \"maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:43.120045 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:43.120004    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l" event={"ID":"474d3de6-0135-47cb-880b-3fae81cc586a","Type":"ContainerStarted","Data":"70a2661f25fc28fff35f1c078755fbcdb9bcd9f384b40ce201d334d7c322adb9"}
Sep 18 16:19:43.120486 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:43.120233    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:43.147930 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:43.147902    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91886522-a6e7-46e7-b2ad-6936049b4eef-maas-api-tls\") pod \"maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:43.150568 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:43.150516    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91886522-a6e7-46e7-b2ad-6936049b4eef-maas-api-tls\") pod \"maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:43.155120 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:43.155078    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l" podStartSLOduration=5.839352955 podStartE2EDuration="6.155062854s" totalImagesPullingTime="315.709899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:43.153434276 +0000 UTC m=+878.100933050" watchObservedRunningTime="2026-09-18 16:19:43.155062854 +0000 UTC m=+878.102561675"
Sep 18 16:19:43.271373 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:43.271346    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:43.572650 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:43.572627    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"]
Sep 18 16:19:43.575168 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:19:43.575133    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91886522_a6e7_46e7_b2ad_6936049b4eef.slice/crio-7c056232080173ea6543da64ea09ff9faae862d9c6fc8a4826680639d5872328 WatchSource:0}: Error finding container 7c056232080173ea6543da64ea09ff9faae862d9c6fc8a4826680639d5872328: Status 404 returned error can't find the container with id 7c056232080173ea6543da64ea09ff9faae862d9c6fc8a4826680639d5872328
Sep 18 16:19:44.127748 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:44.127716    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp" event={"ID":"91886522-a6e7-46e7-b2ad-6936049b4eef","Type":"ContainerStarted","Data":"56d9db1e6077f69c3f1762435c37179ad4f10265aefe27e98d4ece385bd2ad51"}
Sep 18 16:19:44.128139 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:44.127773    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:44.128139 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:44.127790    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp" event={"ID":"91886522-a6e7-46e7-b2ad-6936049b4eef","Type":"ContainerStarted","Data":"7c056232080173ea6543da64ea09ff9faae862d9c6fc8a4826680639d5872328"}
Sep 18 16:19:44.147191 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:44.147056    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp" podStartSLOduration=1.814480544 podStartE2EDuration="2.147038956s" totalImagesPullingTime="332.558412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:44.146420333 +0000 UTC m=+879.093919122" watchObservedRunningTime="2026-09-18 16:19:44.147038956 +0000 UTC m=+879.094537733"
Sep 18 16:19:49.133782 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:49.133708    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:19:50.137587 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:50.137556    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:19:53.122161 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:53.122127    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"
Sep 18 16:19:54.923468 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:54.923435    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"]
Sep 18 16:19:54.927647 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:54.927622    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:54.930318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:54.930294    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f22789-serving-cert\""
Sep 18 16:19:54.937593 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:54.937495    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"]
Sep 18 16:19:54.943355 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:54.943333    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-maas-api-tls\") pod \"maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:54.943433 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:54.943367    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76fea163-46ab-4e45-9b9b-f62073e880ee-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:54.943433 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:54.943386    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-metrics-certs\") pod \"maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:54.943513 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:54.943454    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7chj\" (UniqueName: \"kubernetes.io/projected/76fea163-46ab-4e45-9b9b-f62073e880ee-kube-api-access-n7chj\") pod \"maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:55.044135 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:55.044098    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-maas-api-tls\") pod \"maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:55.044326 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:55.044143    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76fea163-46ab-4e45-9b9b-f62073e880ee-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:55.044326 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:55.044171    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-metrics-certs\") pod \"maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:55.044326 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:55.044222    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7chj\" (UniqueName: \"kubernetes.io/projected/76fea163-46ab-4e45-9b9b-f62073e880ee-kube-api-access-n7chj\") pod \"maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:55.044326 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:19:55.044262    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-serving-cert: secret "maas-api-e2e-inf-b-f22789-serving-cert" not found
Sep 18 16:19:55.044552 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:19:55.044332    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-maas-api-tls podName:76fea163-46ab-4e45-9b9b-f62073e880ee nodeName:}" failed. No retries permitted until 2026-09-18 16:19:55.544312455 +0000 UTC m=+890.491811212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-maas-api-tls") pod "maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z" (UID: "76fea163-46ab-4e45-9b9b-f62073e880ee") : secret "maas-api-e2e-inf-b-f22789-serving-cert" not found
Sep 18 16:19:55.045081 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:55.045007    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76fea163-46ab-4e45-9b9b-f62073e880ee-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:55.046897 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:55.046870    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-metrics-certs\") pod \"maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:55.053146 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:55.053126    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7chj\" (UniqueName: \"kubernetes.io/projected/76fea163-46ab-4e45-9b9b-f62073e880ee-kube-api-access-n7chj\") pod \"maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:55.549292 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:55.549253    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-maas-api-tls\") pod \"maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:55.549456 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:19:55.549400    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-serving-cert: secret "maas-api-e2e-inf-b-f22789-serving-cert" not found
Sep 18 16:19:55.549497 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:19:55.549470    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-maas-api-tls podName:76fea163-46ab-4e45-9b9b-f62073e880ee nodeName:}" failed. No retries permitted until 2026-09-18 16:19:56.549454968 +0000 UTC m=+891.496953720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-maas-api-tls") pod "maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z" (UID: "76fea163-46ab-4e45-9b9b-f62073e880ee") : secret "maas-api-e2e-inf-b-f22789-serving-cert" not found
Sep 18 16:19:56.558897 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:56.558865    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-maas-api-tls\") pod \"maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:56.561368 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:56.561342    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-maas-api-tls\") pod \"maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:56.739341 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:56.739312    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:57.012436 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:57.012413    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"]
Sep 18 16:19:57.014192 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:19:57.014168    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fea163_46ab_4e45_9b9b_f62073e880ee.slice/crio-f8ce5fa2b77004bbfdaa9d26625950c3ff9a9c0ddde9019a1ad4d4d3180dfb27 WatchSource:0}: Error finding container f8ce5fa2b77004bbfdaa9d26625950c3ff9a9c0ddde9019a1ad4d4d3180dfb27: Status 404 returned error can't find the container with id f8ce5fa2b77004bbfdaa9d26625950c3ff9a9c0ddde9019a1ad4d4d3180dfb27
Sep 18 16:19:57.193357 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:57.193329    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z" event={"ID":"76fea163-46ab-4e45-9b9b-f62073e880ee","Type":"ContainerStarted","Data":"f8ce5fa2b77004bbfdaa9d26625950c3ff9a9c0ddde9019a1ad4d4d3180dfb27"}
Sep 18 16:19:58.199760 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:58.199716    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z" event={"ID":"76fea163-46ab-4e45-9b9b-f62073e880ee","Type":"ContainerStarted","Data":"59333b9687dc0527535a5b6f01297effc457be0526333cd86cd064cb55c6abe8"}
Sep 18 16:19:58.200148 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:58.199783    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:19:58.224588 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:19:58.224515    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z" podStartSLOduration=3.649905665 podStartE2EDuration="4.224500641s" totalImagesPullingTime="574.594976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:58.222411179 +0000 UTC m=+893.169909952" watchObservedRunningTime="2026-09-18 16:19:58.224500641 +0000 UTC m=+893.171999414"
Sep 18 16:20:04.098323 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.098290    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"]
Sep 18 16:20:04.098830 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.098549    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp" podUID="91886522-a6e7-46e7-b2ad-6936049b4eef" containerName="maas-api" containerID="cri-o://56d9db1e6077f69c3f1762435c37179ad4f10265aefe27e98d4ece385bd2ad51" gracePeriod=30
Sep 18 16:20:04.210366 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.210340    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:20:04.234705 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.234669    2554 generic.go:356] "Generic (PLEG): container finished" podID="91886522-a6e7-46e7-b2ad-6936049b4eef" containerID="56d9db1e6077f69c3f1762435c37179ad4f10265aefe27e98d4ece385bd2ad51" exitCode=0
Sep 18 16:20:04.234851 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.234736    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp" event={"ID":"91886522-a6e7-46e7-b2ad-6936049b4eef","Type":"ContainerDied","Data":"56d9db1e6077f69c3f1762435c37179ad4f10265aefe27e98d4ece385bd2ad51"}
Sep 18 16:20:04.355800 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.355743    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:20:04.438776 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.438744    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4brs\" (UniqueName: \"kubernetes.io/projected/91886522-a6e7-46e7-b2ad-6936049b4eef-kube-api-access-m4brs\") pod \"91886522-a6e7-46e7-b2ad-6936049b4eef\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") "
Sep 18 16:20:04.438937 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.438789    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91886522-a6e7-46e7-b2ad-6936049b4eef-openshift-service-ca\") pod \"91886522-a6e7-46e7-b2ad-6936049b4eef\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") "
Sep 18 16:20:04.438937 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.438882    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91886522-a6e7-46e7-b2ad-6936049b4eef-metrics-certs\") pod \"91886522-a6e7-46e7-b2ad-6936049b4eef\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") "
Sep 18 16:20:04.438937 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.438934    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91886522-a6e7-46e7-b2ad-6936049b4eef-maas-api-tls\") pod \"91886522-a6e7-46e7-b2ad-6936049b4eef\" (UID: \"91886522-a6e7-46e7-b2ad-6936049b4eef\") "
Sep 18 16:20:04.439207 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.439183    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91886522-a6e7-46e7-b2ad-6936049b4eef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "91886522-a6e7-46e7-b2ad-6936049b4eef" (UID: "91886522-a6e7-46e7-b2ad-6936049b4eef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:20:04.441233 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.441207    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91886522-a6e7-46e7-b2ad-6936049b4eef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "91886522-a6e7-46e7-b2ad-6936049b4eef" (UID: "91886522-a6e7-46e7-b2ad-6936049b4eef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:20:04.441359 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.441241    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91886522-a6e7-46e7-b2ad-6936049b4eef-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "91886522-a6e7-46e7-b2ad-6936049b4eef" (UID: "91886522-a6e7-46e7-b2ad-6936049b4eef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:20:04.441359 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.441317    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91886522-a6e7-46e7-b2ad-6936049b4eef-kube-api-access-m4brs" (OuterVolumeSpecName: "kube-api-access-m4brs") pod "91886522-a6e7-46e7-b2ad-6936049b4eef" (UID: "91886522-a6e7-46e7-b2ad-6936049b4eef"). InnerVolumeSpecName "kube-api-access-m4brs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:20:04.540485 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.540456    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91886522-a6e7-46e7-b2ad-6936049b4eef-metrics-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:20:04.540683 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.540485    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91886522-a6e7-46e7-b2ad-6936049b4eef-maas-api-tls\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:20:04.540683 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.540553    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4brs\" (UniqueName: \"kubernetes.io/projected/91886522-a6e7-46e7-b2ad-6936049b4eef-kube-api-access-m4brs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:20:04.540683 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:04.540568    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91886522-a6e7-46e7-b2ad-6936049b4eef-openshift-service-ca\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:20:05.240802 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:05.240770    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"
Sep 18 16:20:05.240802 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:05.240789    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp" event={"ID":"91886522-a6e7-46e7-b2ad-6936049b4eef","Type":"ContainerDied","Data":"7c056232080173ea6543da64ea09ff9faae862d9c6fc8a4826680639d5872328"}
Sep 18 16:20:05.241327 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:05.240842    2554 scope.go:117] "RemoveContainer" containerID="56d9db1e6077f69c3f1762435c37179ad4f10265aefe27e98d4ece385bd2ad51"
Sep 18 16:20:05.278732 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:05.278700    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"]
Sep 18 16:20:05.288546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:05.288506    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-75688a34-586fd5fcc6-kj7kp"]
Sep 18 16:20:07.276702 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:07.276675    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91886522-a6e7-46e7-b2ad-6936049b4eef" path="/var/lib/kubelet/pods/91886522-a6e7-46e7-b2ad-6936049b4eef/volumes"
Sep 18 16:20:12.687909 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.687877    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"]
Sep 18 16:20:12.688348 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.688272    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91886522-a6e7-46e7-b2ad-6936049b4eef" containerName="maas-api"
Sep 18 16:20:12.688348 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.688283    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="91886522-a6e7-46e7-b2ad-6936049b4eef" containerName="maas-api"
Sep 18 16:20:12.688417 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.688351    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="91886522-a6e7-46e7-b2ad-6936049b4eef" containerName="maas-api"
Sep 18 16:20:12.693017 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.692999    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:12.696121 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.696105    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-50cecd8f-serving-cert\""
Sep 18 16:20:12.714494 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.714461    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"]
Sep 18 16:20:12.811422 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.811384    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6829db-d784-49b2-b8cd-7268f898756c-maas-api-tls\") pod \"maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb\" (UID: \"cc6829db-d784-49b2-b8cd-7268f898756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:12.811619 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.811439    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6829db-d784-49b2-b8cd-7268f898756c-openshift-service-ca\") pod \"maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb\" (UID: \"cc6829db-d784-49b2-b8cd-7268f898756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:12.811619 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.811493    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6829db-d784-49b2-b8cd-7268f898756c-metrics-certs\") pod \"maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb\" (UID: \"cc6829db-d784-49b2-b8cd-7268f898756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:12.811716 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.811637    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48tmj\" (UniqueName: \"kubernetes.io/projected/cc6829db-d784-49b2-b8cd-7268f898756c-kube-api-access-48tmj\") pod \"maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb\" (UID: \"cc6829db-d784-49b2-b8cd-7268f898756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:12.912414 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.912385    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48tmj\" (UniqueName: \"kubernetes.io/projected/cc6829db-d784-49b2-b8cd-7268f898756c-kube-api-access-48tmj\") pod \"maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb\" (UID: \"cc6829db-d784-49b2-b8cd-7268f898756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:12.912628 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.912423    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6829db-d784-49b2-b8cd-7268f898756c-maas-api-tls\") pod \"maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb\" (UID: \"cc6829db-d784-49b2-b8cd-7268f898756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:12.912628 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.912447    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6829db-d784-49b2-b8cd-7268f898756c-openshift-service-ca\") pod \"maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb\" (UID: \"cc6829db-d784-49b2-b8cd-7268f898756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:12.912628 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.912470    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6829db-d784-49b2-b8cd-7268f898756c-metrics-certs\") pod \"maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb\" (UID: \"cc6829db-d784-49b2-b8cd-7268f898756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:12.913241 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.913166    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6829db-d784-49b2-b8cd-7268f898756c-openshift-service-ca\") pod \"maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb\" (UID: \"cc6829db-d784-49b2-b8cd-7268f898756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:12.915088 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.915059    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6829db-d784-49b2-b8cd-7268f898756c-metrics-certs\") pod \"maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb\" (UID: \"cc6829db-d784-49b2-b8cd-7268f898756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:12.915254 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.915230    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6829db-d784-49b2-b8cd-7268f898756c-maas-api-tls\") pod \"maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb\" (UID: \"cc6829db-d784-49b2-b8cd-7268f898756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:12.930750 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:12.930728    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48tmj\" (UniqueName: \"kubernetes.io/projected/cc6829db-d784-49b2-b8cd-7268f898756c-kube-api-access-48tmj\") pod \"maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb\" (UID: \"cc6829db-d784-49b2-b8cd-7268f898756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:13.002797 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:13.002745    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:13.286428 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:13.286405    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"]
Sep 18 16:20:13.287364 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:20:13.287338    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6829db_d784_49b2_b8cd_7268f898756c.slice/crio-b8dea13c98099115c7ba4d8ec6f7d932b444c0e8bd989d5ffd6109f0e8073cad WatchSource:0}: Error finding container b8dea13c98099115c7ba4d8ec6f7d932b444c0e8bd989d5ffd6109f0e8073cad: Status 404 returned error can't find the container with id b8dea13c98099115c7ba4d8ec6f7d932b444c0e8bd989d5ffd6109f0e8073cad
Sep 18 16:20:14.290482 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:14.290442    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb" event={"ID":"cc6829db-d784-49b2-b8cd-7268f898756c","Type":"ContainerStarted","Data":"e15a8a3cc3e28a5f10c2e4d02b5e14bfaeb919a76e36bc9e36f62daab70f5fc7"}
Sep 18 16:20:14.290880 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:14.290493    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb" event={"ID":"cc6829db-d784-49b2-b8cd-7268f898756c","Type":"ContainerStarted","Data":"b8dea13c98099115c7ba4d8ec6f7d932b444c0e8bd989d5ffd6109f0e8073cad"}
Sep 18 16:20:14.290880 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:14.290588    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:14.325562 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:14.325494    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb" podStartSLOduration=1.9607616669999999 podStartE2EDuration="2.32547943s" totalImagesPullingTime="364.717763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:14.325245199 +0000 UTC m=+909.272743992" watchObservedRunningTime="2026-09-18 16:20:14.32547943 +0000 UTC m=+909.272978203"
Sep 18 16:20:20.300480 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:20.300444    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:38.924656 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:38.924623    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"]
Sep 18 16:20:38.925151 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:38.924893    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb" podUID="cc6829db-d784-49b2-b8cd-7268f898756c" containerName="maas-api" containerID="cri-o://e15a8a3cc3e28a5f10c2e4d02b5e14bfaeb919a76e36bc9e36f62daab70f5fc7" gracePeriod=30
Sep 18 16:20:39.183465 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.183405    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:39.255489 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.255462    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6829db-d784-49b2-b8cd-7268f898756c-openshift-service-ca\") pod \"cc6829db-d784-49b2-b8cd-7268f898756c\" (UID: \"cc6829db-d784-49b2-b8cd-7268f898756c\") "
Sep 18 16:20:39.255691 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.255497    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6829db-d784-49b2-b8cd-7268f898756c-metrics-certs\") pod \"cc6829db-d784-49b2-b8cd-7268f898756c\" (UID: \"cc6829db-d784-49b2-b8cd-7268f898756c\") "
Sep 18 16:20:39.255691 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.255581    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6829db-d784-49b2-b8cd-7268f898756c-maas-api-tls\") pod \"cc6829db-d784-49b2-b8cd-7268f898756c\" (UID: \"cc6829db-d784-49b2-b8cd-7268f898756c\") "
Sep 18 16:20:39.255691 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.255609    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48tmj\" (UniqueName: \"kubernetes.io/projected/cc6829db-d784-49b2-b8cd-7268f898756c-kube-api-access-48tmj\") pod \"cc6829db-d784-49b2-b8cd-7268f898756c\" (UID: \"cc6829db-d784-49b2-b8cd-7268f898756c\") "
Sep 18 16:20:39.255877 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.255829    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6829db-d784-49b2-b8cd-7268f898756c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc6829db-d784-49b2-b8cd-7268f898756c" (UID: "cc6829db-d784-49b2-b8cd-7268f898756c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:20:39.258016 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.257991    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6829db-d784-49b2-b8cd-7268f898756c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cc6829db-d784-49b2-b8cd-7268f898756c" (UID: "cc6829db-d784-49b2-b8cd-7268f898756c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:20:39.258126 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.258064    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6829db-d784-49b2-b8cd-7268f898756c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc6829db-d784-49b2-b8cd-7268f898756c" (UID: "cc6829db-d784-49b2-b8cd-7268f898756c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:20:39.258179 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.258148    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6829db-d784-49b2-b8cd-7268f898756c-kube-api-access-48tmj" (OuterVolumeSpecName: "kube-api-access-48tmj") pod "cc6829db-d784-49b2-b8cd-7268f898756c" (UID: "cc6829db-d784-49b2-b8cd-7268f898756c"). InnerVolumeSpecName "kube-api-access-48tmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:20:39.356995 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.356956    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48tmj\" (UniqueName: \"kubernetes.io/projected/cc6829db-d784-49b2-b8cd-7268f898756c-kube-api-access-48tmj\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:20:39.356995 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.356990    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6829db-d784-49b2-b8cd-7268f898756c-openshift-service-ca\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:20:39.357170 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.357004    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6829db-d784-49b2-b8cd-7268f898756c-metrics-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:20:39.357170 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.357016    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6829db-d784-49b2-b8cd-7268f898756c-maas-api-tls\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:20:39.423151 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.423121    2554 generic.go:356] "Generic (PLEG): container finished" podID="cc6829db-d784-49b2-b8cd-7268f898756c" containerID="e15a8a3cc3e28a5f10c2e4d02b5e14bfaeb919a76e36bc9e36f62daab70f5fc7" exitCode=0
Sep 18 16:20:39.423281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.423168    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"
Sep 18 16:20:39.423281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.423212    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb" event={"ID":"cc6829db-d784-49b2-b8cd-7268f898756c","Type":"ContainerDied","Data":"e15a8a3cc3e28a5f10c2e4d02b5e14bfaeb919a76e36bc9e36f62daab70f5fc7"}
Sep 18 16:20:39.423281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.423274    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb" event={"ID":"cc6829db-d784-49b2-b8cd-7268f898756c","Type":"ContainerDied","Data":"b8dea13c98099115c7ba4d8ec6f7d932b444c0e8bd989d5ffd6109f0e8073cad"}
Sep 18 16:20:39.423383 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.423306    2554 scope.go:117] "RemoveContainer" containerID="e15a8a3cc3e28a5f10c2e4d02b5e14bfaeb919a76e36bc9e36f62daab70f5fc7"
Sep 18 16:20:39.432684 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.432663    2554 scope.go:117] "RemoveContainer" containerID="e15a8a3cc3e28a5f10c2e4d02b5e14bfaeb919a76e36bc9e36f62daab70f5fc7"
Sep 18 16:20:39.432940 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:20:39.432913    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e15a8a3cc3e28a5f10c2e4d02b5e14bfaeb919a76e36bc9e36f62daab70f5fc7\": container with ID starting with e15a8a3cc3e28a5f10c2e4d02b5e14bfaeb919a76e36bc9e36f62daab70f5fc7 not found: ID does not exist" containerID="e15a8a3cc3e28a5f10c2e4d02b5e14bfaeb919a76e36bc9e36f62daab70f5fc7"
Sep 18 16:20:39.432989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.432939    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15a8a3cc3e28a5f10c2e4d02b5e14bfaeb919a76e36bc9e36f62daab70f5fc7"} err="failed to get container status \"e15a8a3cc3e28a5f10c2e4d02b5e14bfaeb919a76e36bc9e36f62daab70f5fc7\": rpc error: code = NotFound desc = could not find container \"e15a8a3cc3e28a5f10c2e4d02b5e14bfaeb919a76e36bc9e36f62daab70f5fc7\": container with ID starting with e15a8a3cc3e28a5f10c2e4d02b5e14bfaeb919a76e36bc9e36f62daab70f5fc7 not found: ID does not exist"
Sep 18 16:20:39.453829 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.453773    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"]
Sep 18 16:20:39.491311 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:39.491287    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-50cecd8f-76db8fdf67-th6zb"]
Sep 18 16:20:41.277906 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:41.277872    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc6829db-d784-49b2-b8cd-7268f898756c" path="/var/lib/kubelet/pods/cc6829db-d784-49b2-b8cd-7268f898756c/volumes"
Sep 18 16:20:50.574403 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.574374    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"]
Sep 18 16:20:50.574817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.574801    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc6829db-d784-49b2-b8cd-7268f898756c" containerName="maas-api"
Sep 18 16:20:50.574817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.574813    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc6829db-d784-49b2-b8cd-7268f898756c" containerName="maas-api"
Sep 18 16:20:50.574887 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.574883    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc6829db-d784-49b2-b8cd-7268f898756c" containerName="maas-api"
Sep 18 16:20:50.579231 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.579214    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:20:50.583281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.583262    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-48de7216-serving-cert\""
Sep 18 16:20:50.602692 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.602673    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"]
Sep 18 16:20:50.658221 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.658196    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-metrics-certs\") pod \"maas-api-e2e-ait-48de7216-7548864f9c-zmwbc\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:20:50.658353 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.658249    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-maas-api-tls\") pod \"maas-api-e2e-ait-48de7216-7548864f9c-zmwbc\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:20:50.658353 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.658333    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-openshift-service-ca\") pod \"maas-api-e2e-ait-48de7216-7548864f9c-zmwbc\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:20:50.658429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.658386    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvdzt\" (UniqueName: \"kubernetes.io/projected/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-kube-api-access-kvdzt\") pod \"maas-api-e2e-ait-48de7216-7548864f9c-zmwbc\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:20:50.759670 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.759637    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvdzt\" (UniqueName: \"kubernetes.io/projected/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-kube-api-access-kvdzt\") pod \"maas-api-e2e-ait-48de7216-7548864f9c-zmwbc\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:20:50.759813 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.759681    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-metrics-certs\") pod \"maas-api-e2e-ait-48de7216-7548864f9c-zmwbc\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:20:50.759813 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.759727    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-maas-api-tls\") pod \"maas-api-e2e-ait-48de7216-7548864f9c-zmwbc\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:20:50.759813 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.759761    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-openshift-service-ca\") pod \"maas-api-e2e-ait-48de7216-7548864f9c-zmwbc\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:20:50.759918 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:20:50.759858    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-serving-cert: secret "maas-api-e2e-ait-48de7216-serving-cert" not found
Sep 18 16:20:50.759955 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:20:50.759927    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-maas-api-tls podName:93f83f48-1414-4674-bfdf-6cc1dfeaf67c nodeName:}" failed. No retries permitted until 2026-09-18 16:20:51.259908414 +0000 UTC m=+946.207407179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-maas-api-tls") pod "maas-api-e2e-ait-48de7216-7548864f9c-zmwbc" (UID: "93f83f48-1414-4674-bfdf-6cc1dfeaf67c") : secret "maas-api-e2e-ait-48de7216-serving-cert" not found
Sep 18 16:20:50.760320 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.760293    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-openshift-service-ca\") pod \"maas-api-e2e-ait-48de7216-7548864f9c-zmwbc\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:20:50.762218 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.762194    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-metrics-certs\") pod \"maas-api-e2e-ait-48de7216-7548864f9c-zmwbc\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:20:50.782349 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:50.782326    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvdzt\" (UniqueName: \"kubernetes.io/projected/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-kube-api-access-kvdzt\") pod \"maas-api-e2e-ait-48de7216-7548864f9c-zmwbc\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:20:51.265098 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:51.265061    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-maas-api-tls\") pod \"maas-api-e2e-ait-48de7216-7548864f9c-zmwbc\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:20:51.267585 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:51.267562    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-maas-api-tls\") pod \"maas-api-e2e-ait-48de7216-7548864f9c-zmwbc\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:20:51.489498 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:51.489466    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:20:51.789199 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:51.789036    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"]
Sep 18 16:20:51.792301 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:20:51.792271    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f83f48_1414_4674_bfdf_6cc1dfeaf67c.slice/crio-80f4c5a87fb915e9051e681520ae1a9b6ed76871a9bccf4256c35899706c482a WatchSource:0}: Error finding container 80f4c5a87fb915e9051e681520ae1a9b6ed76871a9bccf4256c35899706c482a: Status 404 returned error can't find the container with id 80f4c5a87fb915e9051e681520ae1a9b6ed76871a9bccf4256c35899706c482a
Sep 18 16:20:51.961392 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:51.961359    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"]
Sep 18 16:20:51.965108 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:51.965083    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"
Sep 18 16:20:51.968240 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:51.968221    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-48de7216-dockercfg-kmd5m\""
Sep 18 16:20:51.968356 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:51.968281    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-48de7216\""
Sep 18 16:20:51.995400 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:51.995374    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"]
Sep 18 16:20:51.997468 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:51.997446    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"]
Sep 18 16:20:52.001759 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.001739    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"
Sep 18 16:20:52.021973 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.021946    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"]
Sep 18 16:20:52.072385 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.072318    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c070e842-b22c-43c1-8efa-5421957a484b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p\" (UID: \"c070e842-b22c-43c1-8efa-5421957a484b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"
Sep 18 16:20:52.072544 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.072400    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpfgh\" (UniqueName: \"kubernetes.io/projected/c070e842-b22c-43c1-8efa-5421957a484b-kube-api-access-rpfgh\") pod \"payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p\" (UID: \"c070e842-b22c-43c1-8efa-5421957a484b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"
Sep 18 16:20:52.072544 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.072431    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77zg6\" (UniqueName: \"kubernetes.io/projected/6644b6ee-dd54-4ac7-a33a-34ca6075c87f-kube-api-access-77zg6\") pod \"payload-processing-e2e-ait-48de7216-cf4874f57-65d6c\" (UID: \"6644b6ee-dd54-4ac7-a33a-34ca6075c87f\") " pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"
Sep 18 16:20:52.072544 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.072469    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6644b6ee-dd54-4ac7-a33a-34ca6075c87f-plugins-config-volume\") pod \"payload-processing-e2e-ait-48de7216-cf4874f57-65d6c\" (UID: \"6644b6ee-dd54-4ac7-a33a-34ca6075c87f\") " pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"
Sep 18 16:20:52.173920 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.173879    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c070e842-b22c-43c1-8efa-5421957a484b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p\" (UID: \"c070e842-b22c-43c1-8efa-5421957a484b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"
Sep 18 16:20:52.174126 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.173971    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpfgh\" (UniqueName: \"kubernetes.io/projected/c070e842-b22c-43c1-8efa-5421957a484b-kube-api-access-rpfgh\") pod \"payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p\" (UID: \"c070e842-b22c-43c1-8efa-5421957a484b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"
Sep 18 16:20:52.174126 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.174002    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77zg6\" (UniqueName: \"kubernetes.io/projected/6644b6ee-dd54-4ac7-a33a-34ca6075c87f-kube-api-access-77zg6\") pod \"payload-processing-e2e-ait-48de7216-cf4874f57-65d6c\" (UID: \"6644b6ee-dd54-4ac7-a33a-34ca6075c87f\") " pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"
Sep 18 16:20:52.174126 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.174041    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6644b6ee-dd54-4ac7-a33a-34ca6075c87f-plugins-config-volume\") pod \"payload-processing-e2e-ait-48de7216-cf4874f57-65d6c\" (UID: \"6644b6ee-dd54-4ac7-a33a-34ca6075c87f\") " pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"
Sep 18 16:20:52.174737 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.174713    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c070e842-b22c-43c1-8efa-5421957a484b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p\" (UID: \"c070e842-b22c-43c1-8efa-5421957a484b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"
Sep 18 16:20:52.174838 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.174778    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6644b6ee-dd54-4ac7-a33a-34ca6075c87f-plugins-config-volume\") pod \"payload-processing-e2e-ait-48de7216-cf4874f57-65d6c\" (UID: \"6644b6ee-dd54-4ac7-a33a-34ca6075c87f\") " pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"
Sep 18 16:20:52.194146 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.194123    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77zg6\" (UniqueName: \"kubernetes.io/projected/6644b6ee-dd54-4ac7-a33a-34ca6075c87f-kube-api-access-77zg6\") pod \"payload-processing-e2e-ait-48de7216-cf4874f57-65d6c\" (UID: \"6644b6ee-dd54-4ac7-a33a-34ca6075c87f\") " pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"
Sep 18 16:20:52.198104 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.198072    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpfgh\" (UniqueName: \"kubernetes.io/projected/c070e842-b22c-43c1-8efa-5421957a484b-kube-api-access-rpfgh\") pod \"payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p\" (UID: \"c070e842-b22c-43c1-8efa-5421957a484b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"
Sep 18 16:20:52.276233 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.276209    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"
Sep 18 16:20:52.312213 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.312185    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"
Sep 18 16:20:52.490507 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.490470    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc" event={"ID":"93f83f48-1414-4674-bfdf-6cc1dfeaf67c","Type":"ContainerStarted","Data":"657ad67b5a2cc555da7ce429aa5b5fbed3c5a72e072ede6df3f562674918903e"}
Sep 18 16:20:52.490715 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.490537    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc" event={"ID":"93f83f48-1414-4674-bfdf-6cc1dfeaf67c","Type":"ContainerStarted","Data":"80f4c5a87fb915e9051e681520ae1a9b6ed76871a9bccf4256c35899706c482a"}
Sep 18 16:20:52.490715 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.490684    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:20:52.599051 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:20:52.599020    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc070e842_b22c_43c1_8efa_5421957a484b.slice/crio-2d3ef1df88f9100ed136a19631df0906024ef72f71b644748de1454fbad22c65 WatchSource:0}: Error finding container 2d3ef1df88f9100ed136a19631df0906024ef72f71b644748de1454fbad22c65: Status 404 returned error can't find the container with id 2d3ef1df88f9100ed136a19631df0906024ef72f71b644748de1454fbad22c65
Sep 18 16:20:52.602978 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.602934    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc" podStartSLOduration=2.12291466 podStartE2EDuration="2.60291915s" totalImagesPullingTime="480.00449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:52.577651943 +0000 UTC m=+947.525150722" watchObservedRunningTime="2026-09-18 16:20:52.60291915 +0000 UTC m=+947.550417924"
Sep 18 16:20:52.604402 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.604383    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"]
Sep 18 16:20:52.644612 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:52.644589    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"]
Sep 18 16:20:52.646184 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:20:52.646158    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6644b6ee_dd54_4ac7_a33a_34ca6075c87f.slice/crio-78b43cf596b1300d9224c17f7ae2c3bde6a446a3d3cd959a12738c8f7cadbec1 WatchSource:0}: Error finding container 78b43cf596b1300d9224c17f7ae2c3bde6a446a3d3cd959a12738c8f7cadbec1: Status 404 returned error can't find the container with id 78b43cf596b1300d9224c17f7ae2c3bde6a446a3d3cd959a12738c8f7cadbec1
Sep 18 16:20:53.497783 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:53.497748    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c" event={"ID":"6644b6ee-dd54-4ac7-a33a-34ca6075c87f","Type":"ContainerStarted","Data":"71755f781adfb85156ac35decdce65f005e0d613a24821c962b90e85fc805e78"}
Sep 18 16:20:53.498259 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:53.497883    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"
Sep 18 16:20:53.498259 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:53.497946    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c" event={"ID":"6644b6ee-dd54-4ac7-a33a-34ca6075c87f","Type":"ContainerStarted","Data":"78b43cf596b1300d9224c17f7ae2c3bde6a446a3d3cd959a12738c8f7cadbec1"}
Sep 18 16:20:53.499421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:53.499395    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p" event={"ID":"c070e842-b22c-43c1-8efa-5421957a484b","Type":"ContainerStarted","Data":"af9a483585a10fa15775f10250e101df7245440ac723df9c798cf4af21c8bac5"}
Sep 18 16:20:53.499560 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:53.499439    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p" event={"ID":"c070e842-b22c-43c1-8efa-5421957a484b","Type":"ContainerStarted","Data":"2d3ef1df88f9100ed136a19631df0906024ef72f71b644748de1454fbad22c65"}
Sep 18 16:20:53.499654 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:53.499638    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"
Sep 18 16:20:53.564405 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:53.564346    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p" podStartSLOduration=2.564325963 podStartE2EDuration="2.564325963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:53.563918808 +0000 UTC m=+948.511417579" watchObservedRunningTime="2026-09-18 16:20:53.564325963 +0000 UTC m=+948.511824741"
Sep 18 16:20:53.565138 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:53.565102    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c" podStartSLOduration=2.565091475 podStartE2EDuration="2.565091475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:53.533565449 +0000 UTC m=+948.481064223" watchObservedRunningTime="2026-09-18 16:20:53.565091475 +0000 UTC m=+948.512590297"
Sep 18 16:20:58.505428 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:20:58.505395    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:21:04.506576 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:04.506543    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"
Sep 18 16:21:04.506970 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:04.506630    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"
Sep 18 16:21:34.970603 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:34.970503    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"]
Sep 18 16:21:34.971098 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:34.970745    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc" podUID="93f83f48-1414-4674-bfdf-6cc1dfeaf67c" containerName="maas-api" containerID="cri-o://657ad67b5a2cc555da7ce429aa5b5fbed3c5a72e072ede6df3f562674918903e" gracePeriod=30
Sep 18 16:21:35.055278 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.055245    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"]
Sep 18 16:21:35.055486 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.055449    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c" podUID="6644b6ee-dd54-4ac7-a33a-34ca6075c87f" containerName="payload-processing" containerID="cri-o://71755f781adfb85156ac35decdce65f005e0d613a24821c962b90e85fc805e78" gracePeriod=30
Sep 18 16:21:35.082393 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.082362    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"]
Sep 18 16:21:35.082988 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.082958    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p" podUID="c070e842-b22c-43c1-8efa-5421957a484b" containerName="payload-pre-processing" containerID="cri-o://af9a483585a10fa15775f10250e101df7245440ac723df9c798cf4af21c8bac5" gracePeriod=30
Sep 18 16:21:35.319247 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.319216    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:21:35.368114 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.368028    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"
Sep 18 16:21:35.375228 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.375204    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"
Sep 18 16:21:35.380193 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.380171    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-maas-api-tls\") pod \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") "
Sep 18 16:21:35.380316 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.380213    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-openshift-service-ca\") pod \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") "
Sep 18 16:21:35.380316 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.380233    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-metrics-certs\") pod \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") "
Sep 18 16:21:35.380316 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.380301    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvdzt\" (UniqueName: \"kubernetes.io/projected/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-kube-api-access-kvdzt\") pod \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\" (UID: \"93f83f48-1414-4674-bfdf-6cc1dfeaf67c\") "
Sep 18 16:21:35.380591 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.380563    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93f83f48-1414-4674-bfdf-6cc1dfeaf67c" (UID: "93f83f48-1414-4674-bfdf-6cc1dfeaf67c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:21:35.382495 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.382472    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "93f83f48-1414-4674-bfdf-6cc1dfeaf67c" (UID: "93f83f48-1414-4674-bfdf-6cc1dfeaf67c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:21:35.382631 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.382472    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-kube-api-access-kvdzt" (OuterVolumeSpecName: "kube-api-access-kvdzt") pod "93f83f48-1414-4674-bfdf-6cc1dfeaf67c" (UID: "93f83f48-1414-4674-bfdf-6cc1dfeaf67c"). InnerVolumeSpecName "kube-api-access-kvdzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:21:35.382631 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.382496    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93f83f48-1414-4674-bfdf-6cc1dfeaf67c" (UID: "93f83f48-1414-4674-bfdf-6cc1dfeaf67c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:21:35.481318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.481220    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6644b6ee-dd54-4ac7-a33a-34ca6075c87f-plugins-config-volume\") pod \"6644b6ee-dd54-4ac7-a33a-34ca6075c87f\" (UID: \"6644b6ee-dd54-4ac7-a33a-34ca6075c87f\") "
Sep 18 16:21:35.481318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.481297    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpfgh\" (UniqueName: \"kubernetes.io/projected/c070e842-b22c-43c1-8efa-5421957a484b-kube-api-access-rpfgh\") pod \"c070e842-b22c-43c1-8efa-5421957a484b\" (UID: \"c070e842-b22c-43c1-8efa-5421957a484b\") "
Sep 18 16:21:35.481505 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.481327    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77zg6\" (UniqueName: \"kubernetes.io/projected/6644b6ee-dd54-4ac7-a33a-34ca6075c87f-kube-api-access-77zg6\") pod \"6644b6ee-dd54-4ac7-a33a-34ca6075c87f\" (UID: \"6644b6ee-dd54-4ac7-a33a-34ca6075c87f\") "
Sep 18 16:21:35.481505 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.481383    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c070e842-b22c-43c1-8efa-5421957a484b-plugins-config-volume\") pod \"c070e842-b22c-43c1-8efa-5421957a484b\" (UID: \"c070e842-b22c-43c1-8efa-5421957a484b\") "
Sep 18 16:21:35.481827 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.481674    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6644b6ee-dd54-4ac7-a33a-34ca6075c87f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6644b6ee-dd54-4ac7-a33a-34ca6075c87f" (UID: "6644b6ee-dd54-4ac7-a33a-34ca6075c87f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:21:35.481944 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.481745    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-metrics-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:21:35.481944 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.481863    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvdzt\" (UniqueName: \"kubernetes.io/projected/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-kube-api-access-kvdzt\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:21:35.481944 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.481881    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-maas-api-tls\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:21:35.481944 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.481896    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f83f48-1414-4674-bfdf-6cc1dfeaf67c-openshift-service-ca\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:21:35.481944 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.481861    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c070e842-b22c-43c1-8efa-5421957a484b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c070e842-b22c-43c1-8efa-5421957a484b" (UID: "c070e842-b22c-43c1-8efa-5421957a484b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:21:35.483826 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.483804    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6644b6ee-dd54-4ac7-a33a-34ca6075c87f-kube-api-access-77zg6" (OuterVolumeSpecName: "kube-api-access-77zg6") pod "6644b6ee-dd54-4ac7-a33a-34ca6075c87f" (UID: "6644b6ee-dd54-4ac7-a33a-34ca6075c87f"). InnerVolumeSpecName "kube-api-access-77zg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:21:35.483898 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.483804    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c070e842-b22c-43c1-8efa-5421957a484b-kube-api-access-rpfgh" (OuterVolumeSpecName: "kube-api-access-rpfgh") pod "c070e842-b22c-43c1-8efa-5421957a484b" (UID: "c070e842-b22c-43c1-8efa-5421957a484b"). InnerVolumeSpecName "kube-api-access-rpfgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:21:35.582476 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.582420    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6644b6ee-dd54-4ac7-a33a-34ca6075c87f-plugins-config-volume\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:21:35.582476 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.582467    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpfgh\" (UniqueName: \"kubernetes.io/projected/c070e842-b22c-43c1-8efa-5421957a484b-kube-api-access-rpfgh\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:21:35.582476 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.582476    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77zg6\" (UniqueName: \"kubernetes.io/projected/6644b6ee-dd54-4ac7-a33a-34ca6075c87f-kube-api-access-77zg6\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:21:35.582476 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.582487    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c070e842-b22c-43c1-8efa-5421957a484b-plugins-config-volume\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:21:35.719910 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.719875    2554 generic.go:356] "Generic (PLEG): container finished" podID="93f83f48-1414-4674-bfdf-6cc1dfeaf67c" containerID="657ad67b5a2cc555da7ce429aa5b5fbed3c5a72e072ede6df3f562674918903e" exitCode=0
Sep 18 16:21:35.720093 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.719923    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"
Sep 18 16:21:35.720093 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.719964    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc" event={"ID":"93f83f48-1414-4674-bfdf-6cc1dfeaf67c","Type":"ContainerDied","Data":"657ad67b5a2cc555da7ce429aa5b5fbed3c5a72e072ede6df3f562674918903e"}
Sep 18 16:21:35.720093 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.720018    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc" event={"ID":"93f83f48-1414-4674-bfdf-6cc1dfeaf67c","Type":"ContainerDied","Data":"80f4c5a87fb915e9051e681520ae1a9b6ed76871a9bccf4256c35899706c482a"}
Sep 18 16:21:35.720093 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.720046    2554 scope.go:117] "RemoveContainer" containerID="657ad67b5a2cc555da7ce429aa5b5fbed3c5a72e072ede6df3f562674918903e"
Sep 18 16:21:35.721431 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.721409    2554 generic.go:356] "Generic (PLEG): container finished" podID="6644b6ee-dd54-4ac7-a33a-34ca6075c87f" containerID="71755f781adfb85156ac35decdce65f005e0d613a24821c962b90e85fc805e78" exitCode=0
Sep 18 16:21:35.721558 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.721448    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"
Sep 18 16:21:35.721558 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.721515    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c" event={"ID":"6644b6ee-dd54-4ac7-a33a-34ca6075c87f","Type":"ContainerDied","Data":"71755f781adfb85156ac35decdce65f005e0d613a24821c962b90e85fc805e78"}
Sep 18 16:21:35.721798 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.721569    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c" event={"ID":"6644b6ee-dd54-4ac7-a33a-34ca6075c87f","Type":"ContainerDied","Data":"78b43cf596b1300d9224c17f7ae2c3bde6a446a3d3cd959a12738c8f7cadbec1"}
Sep 18 16:21:35.723200 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.723176    2554 generic.go:356] "Generic (PLEG): container finished" podID="c070e842-b22c-43c1-8efa-5421957a484b" containerID="af9a483585a10fa15775f10250e101df7245440ac723df9c798cf4af21c8bac5" exitCode=0
Sep 18 16:21:35.723316 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.723209    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"
Sep 18 16:21:35.723316 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.723211    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p" event={"ID":"c070e842-b22c-43c1-8efa-5421957a484b","Type":"ContainerDied","Data":"af9a483585a10fa15775f10250e101df7245440ac723df9c798cf4af21c8bac5"}
Sep 18 16:21:35.723427 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.723336    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p" event={"ID":"c070e842-b22c-43c1-8efa-5421957a484b","Type":"ContainerDied","Data":"2d3ef1df88f9100ed136a19631df0906024ef72f71b644748de1454fbad22c65"}
Sep 18 16:21:35.732868 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.732467    2554 scope.go:117] "RemoveContainer" containerID="657ad67b5a2cc555da7ce429aa5b5fbed3c5a72e072ede6df3f562674918903e"
Sep 18 16:21:35.733764 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:21:35.733738    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657ad67b5a2cc555da7ce429aa5b5fbed3c5a72e072ede6df3f562674918903e\": container with ID starting with 657ad67b5a2cc555da7ce429aa5b5fbed3c5a72e072ede6df3f562674918903e not found: ID does not exist" containerID="657ad67b5a2cc555da7ce429aa5b5fbed3c5a72e072ede6df3f562674918903e"
Sep 18 16:21:35.733865 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.733774    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657ad67b5a2cc555da7ce429aa5b5fbed3c5a72e072ede6df3f562674918903e"} err="failed to get container status \"657ad67b5a2cc555da7ce429aa5b5fbed3c5a72e072ede6df3f562674918903e\": rpc error: code = NotFound desc = could not find container \"657ad67b5a2cc555da7ce429aa5b5fbed3c5a72e072ede6df3f562674918903e\": container with ID starting with 657ad67b5a2cc555da7ce429aa5b5fbed3c5a72e072ede6df3f562674918903e not found: ID does not exist"
Sep 18 16:21:35.733865 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.733800    2554 scope.go:117] "RemoveContainer" containerID="71755f781adfb85156ac35decdce65f005e0d613a24821c962b90e85fc805e78"
Sep 18 16:21:35.742723 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.742705    2554 scope.go:117] "RemoveContainer" containerID="71755f781adfb85156ac35decdce65f005e0d613a24821c962b90e85fc805e78"
Sep 18 16:21:35.742971 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:21:35.742955    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71755f781adfb85156ac35decdce65f005e0d613a24821c962b90e85fc805e78\": container with ID starting with 71755f781adfb85156ac35decdce65f005e0d613a24821c962b90e85fc805e78 not found: ID does not exist" containerID="71755f781adfb85156ac35decdce65f005e0d613a24821c962b90e85fc805e78"
Sep 18 16:21:35.743046 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.742978    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71755f781adfb85156ac35decdce65f005e0d613a24821c962b90e85fc805e78"} err="failed to get container status \"71755f781adfb85156ac35decdce65f005e0d613a24821c962b90e85fc805e78\": rpc error: code = NotFound desc = could not find container \"71755f781adfb85156ac35decdce65f005e0d613a24821c962b90e85fc805e78\": container with ID starting with 71755f781adfb85156ac35decdce65f005e0d613a24821c962b90e85fc805e78 not found: ID does not exist"
Sep 18 16:21:35.743046 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.742992    2554 scope.go:117] "RemoveContainer" containerID="af9a483585a10fa15775f10250e101df7245440ac723df9c798cf4af21c8bac5"
Sep 18 16:21:35.750862 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.750842    2554 scope.go:117] "RemoveContainer" containerID="af9a483585a10fa15775f10250e101df7245440ac723df9c798cf4af21c8bac5"
Sep 18 16:21:35.751136 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:21:35.751117    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9a483585a10fa15775f10250e101df7245440ac723df9c798cf4af21c8bac5\": container with ID starting with af9a483585a10fa15775f10250e101df7245440ac723df9c798cf4af21c8bac5 not found: ID does not exist" containerID="af9a483585a10fa15775f10250e101df7245440ac723df9c798cf4af21c8bac5"
Sep 18 16:21:35.751188 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.751145    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9a483585a10fa15775f10250e101df7245440ac723df9c798cf4af21c8bac5"} err="failed to get container status \"af9a483585a10fa15775f10250e101df7245440ac723df9c798cf4af21c8bac5\": rpc error: code = NotFound desc = could not find container \"af9a483585a10fa15775f10250e101df7245440ac723df9c798cf4af21c8bac5\": container with ID starting with af9a483585a10fa15775f10250e101df7245440ac723df9c798cf4af21c8bac5 not found: ID does not exist"
Sep 18 16:21:35.754243 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.754221    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"]
Sep 18 16:21:35.759778 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.759757    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-48de7216-fbc4c9874-clh7p"]
Sep 18 16:21:35.783941 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.783914    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"]
Sep 18 16:21:35.788532 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.788493    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-48de7216-7548864f9c-zmwbc"]
Sep 18 16:21:35.818435 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.818399    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"]
Sep 18 16:21:35.821419 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:35.821392    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-48de7216-cf4874f57-65d6c"]
Sep 18 16:21:37.278009 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:37.277974    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6644b6ee-dd54-4ac7-a33a-34ca6075c87f" path="/var/lib/kubelet/pods/6644b6ee-dd54-4ac7-a33a-34ca6075c87f/volumes"
Sep 18 16:21:37.278578 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:37.278558    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93f83f48-1414-4674-bfdf-6cc1dfeaf67c" path="/var/lib/kubelet/pods/93f83f48-1414-4674-bfdf-6cc1dfeaf67c/volumes"
Sep 18 16:21:37.278963 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:37.278945    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c070e842-b22c-43c1-8efa-5421957a484b" path="/var/lib/kubelet/pods/c070e842-b22c-43c1-8efa-5421957a484b/volumes"
Sep 18 16:21:45.212511 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.212477    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"]
Sep 18 16:21:45.212996 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.212889    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6644b6ee-dd54-4ac7-a33a-34ca6075c87f" containerName="payload-processing"
Sep 18 16:21:45.212996 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.212901    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6644b6ee-dd54-4ac7-a33a-34ca6075c87f" containerName="payload-processing"
Sep 18 16:21:45.212996 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.212912    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c070e842-b22c-43c1-8efa-5421957a484b" containerName="payload-pre-processing"
Sep 18 16:21:45.212996 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.212918    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c070e842-b22c-43c1-8efa-5421957a484b" containerName="payload-pre-processing"
Sep 18 16:21:45.212996 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.212932    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93f83f48-1414-4674-bfdf-6cc1dfeaf67c" containerName="maas-api"
Sep 18 16:21:45.212996 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.212938    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="93f83f48-1414-4674-bfdf-6cc1dfeaf67c" containerName="maas-api"
Sep 18 16:21:45.212996 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.212997    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="93f83f48-1414-4674-bfdf-6cc1dfeaf67c" containerName="maas-api"
Sep 18 16:21:45.213373 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.213008    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c070e842-b22c-43c1-8efa-5421957a484b" containerName="payload-pre-processing"
Sep 18 16:21:45.213373 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.213016    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6644b6ee-dd54-4ac7-a33a-34ca6075c87f" containerName="payload-processing"
Sep 18 16:21:45.217881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.217861    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:21:45.223473 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.223446    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-11860c1b-serving-cert\""
Sep 18 16:21:45.245086 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.245060    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"]
Sep 18 16:21:45.379026 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.378989    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcaca9a8-f709-4752-b5eb-42065591f34e-maas-api-tls\") pod \"maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z\" (UID: \"bcaca9a8-f709-4752-b5eb-42065591f34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:21:45.379205 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.379081    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcfgd\" (UniqueName: \"kubernetes.io/projected/bcaca9a8-f709-4752-b5eb-42065591f34e-kube-api-access-zcfgd\") pod \"maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z\" (UID: \"bcaca9a8-f709-4752-b5eb-42065591f34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:21:45.379205 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.379172    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcaca9a8-f709-4752-b5eb-42065591f34e-openshift-service-ca\") pod \"maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z\" (UID: \"bcaca9a8-f709-4752-b5eb-42065591f34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:21:45.379294 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.379230    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcaca9a8-f709-4752-b5eb-42065591f34e-metrics-certs\") pod \"maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z\" (UID: \"bcaca9a8-f709-4752-b5eb-42065591f34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:21:45.480491 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.480408    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcfgd\" (UniqueName: \"kubernetes.io/projected/bcaca9a8-f709-4752-b5eb-42065591f34e-kube-api-access-zcfgd\") pod \"maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z\" (UID: \"bcaca9a8-f709-4752-b5eb-42065591f34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:21:45.480491 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.480469    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcaca9a8-f709-4752-b5eb-42065591f34e-openshift-service-ca\") pod \"maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z\" (UID: \"bcaca9a8-f709-4752-b5eb-42065591f34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:21:45.480729 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.480504    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcaca9a8-f709-4752-b5eb-42065591f34e-metrics-certs\") pod \"maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z\" (UID: \"bcaca9a8-f709-4752-b5eb-42065591f34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:21:45.480729 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.480550    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcaca9a8-f709-4752-b5eb-42065591f34e-maas-api-tls\") pod \"maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z\" (UID: \"bcaca9a8-f709-4752-b5eb-42065591f34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:21:45.481233 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.481205    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcaca9a8-f709-4752-b5eb-42065591f34e-openshift-service-ca\") pod \"maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z\" (UID: \"bcaca9a8-f709-4752-b5eb-42065591f34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:21:45.483151 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.483129    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcaca9a8-f709-4752-b5eb-42065591f34e-maas-api-tls\") pod \"maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z\" (UID: \"bcaca9a8-f709-4752-b5eb-42065591f34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:21:45.483247 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.483206    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcaca9a8-f709-4752-b5eb-42065591f34e-metrics-certs\") pod \"maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z\" (UID: \"bcaca9a8-f709-4752-b5eb-42065591f34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:21:45.496839 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.496808    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcfgd\" (UniqueName: \"kubernetes.io/projected/bcaca9a8-f709-4752-b5eb-42065591f34e-kube-api-access-zcfgd\") pod \"maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z\" (UID: \"bcaca9a8-f709-4752-b5eb-42065591f34e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:21:45.529878 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.529846    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:21:45.814162 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.814084    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"]
Sep 18 16:21:45.816653 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:21:45.816624    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcaca9a8_f709_4752_b5eb_42065591f34e.slice/crio-ad9829816e9b903c8b77e534501c923277ff6ff8e1ef20799cc4fdc9754348fe WatchSource:0}: Error finding container ad9829816e9b903c8b77e534501c923277ff6ff8e1ef20799cc4fdc9754348fe: Status 404 returned error can't find the container with id ad9829816e9b903c8b77e534501c923277ff6ff8e1ef20799cc4fdc9754348fe
Sep 18 16:21:45.817926 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:45.817906    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:21:46.343012 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.342978    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"]
Sep 18 16:21:46.346569 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.346551    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"
Sep 18 16:21:46.351555 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.351509    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-11860c1b\""
Sep 18 16:21:46.351751 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.351604    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-11860c1b-dockercfg-pn7q7\""
Sep 18 16:21:46.361370 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.361347    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"]
Sep 18 16:21:46.445384 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.445348    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"]
Sep 18 16:21:46.448815 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.448788    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"
Sep 18 16:21:46.462923 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.462896    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"]
Sep 18 16:21:46.490299 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.490264    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4df144df-c745-4b79-b4f9-404e5d5cb36c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj\" (UID: \"4df144df-c745-4b79-b4f9-404e5d5cb36c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"
Sep 18 16:21:46.490469 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.490386    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nppjr\" (UniqueName: \"kubernetes.io/projected/4df144df-c745-4b79-b4f9-404e5d5cb36c-kube-api-access-nppjr\") pod \"payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj\" (UID: \"4df144df-c745-4b79-b4f9-404e5d5cb36c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"
Sep 18 16:21:46.591879 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.591783    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8qjh\" (UniqueName: \"kubernetes.io/projected/19d37ff3-1220-4f65-a897-1237beeee2e6-kube-api-access-c8qjh\") pod \"payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94\" (UID: \"19d37ff3-1220-4f65-a897-1237beeee2e6\") " pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"
Sep 18 16:21:46.591879 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.591848    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4df144df-c745-4b79-b4f9-404e5d5cb36c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj\" (UID: \"4df144df-c745-4b79-b4f9-404e5d5cb36c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"
Sep 18 16:21:46.592088 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.591915    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nppjr\" (UniqueName: \"kubernetes.io/projected/4df144df-c745-4b79-b4f9-404e5d5cb36c-kube-api-access-nppjr\") pod \"payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj\" (UID: \"4df144df-c745-4b79-b4f9-404e5d5cb36c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"
Sep 18 16:21:46.592088 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.591943    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d37ff3-1220-4f65-a897-1237beeee2e6-plugins-config-volume\") pod \"payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94\" (UID: \"19d37ff3-1220-4f65-a897-1237beeee2e6\") " pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"
Sep 18 16:21:46.592751 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.592725    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4df144df-c745-4b79-b4f9-404e5d5cb36c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj\" (UID: \"4df144df-c745-4b79-b4f9-404e5d5cb36c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"
Sep 18 16:21:46.602426 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.602395    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nppjr\" (UniqueName: \"kubernetes.io/projected/4df144df-c745-4b79-b4f9-404e5d5cb36c-kube-api-access-nppjr\") pod \"payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj\" (UID: \"4df144df-c745-4b79-b4f9-404e5d5cb36c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"
Sep 18 16:21:46.656721 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.656683    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"
Sep 18 16:21:46.692461 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.692423    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d37ff3-1220-4f65-a897-1237beeee2e6-plugins-config-volume\") pod \"payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94\" (UID: \"19d37ff3-1220-4f65-a897-1237beeee2e6\") " pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"
Sep 18 16:21:46.692651 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.692486    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8qjh\" (UniqueName: \"kubernetes.io/projected/19d37ff3-1220-4f65-a897-1237beeee2e6-kube-api-access-c8qjh\") pod \"payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94\" (UID: \"19d37ff3-1220-4f65-a897-1237beeee2e6\") " pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"
Sep 18 16:21:46.693184 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.693159    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d37ff3-1220-4f65-a897-1237beeee2e6-plugins-config-volume\") pod \"payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94\" (UID: \"19d37ff3-1220-4f65-a897-1237beeee2e6\") " pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"
Sep 18 16:21:46.703127 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.703100    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8qjh\" (UniqueName: \"kubernetes.io/projected/19d37ff3-1220-4f65-a897-1237beeee2e6-kube-api-access-c8qjh\") pod \"payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94\" (UID: \"19d37ff3-1220-4f65-a897-1237beeee2e6\") " pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"
Sep 18 16:21:46.760383 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.760353    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"
Sep 18 16:21:46.784474 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.784436    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z" event={"ID":"bcaca9a8-f709-4752-b5eb-42065591f34e","Type":"ContainerStarted","Data":"d09daa24a4695dba4739d264f726d2fe9913245ef3e1a75337f8f09e7f7fdbec"}
Sep 18 16:21:46.784640 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.784496    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z" event={"ID":"bcaca9a8-f709-4752-b5eb-42065591f34e","Type":"ContainerStarted","Data":"ad9829816e9b903c8b77e534501c923277ff6ff8e1ef20799cc4fdc9754348fe"}
Sep 18 16:21:46.784640 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.784573    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:21:46.822776 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.822493    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z" podStartSLOduration=1.3988016110000001 podStartE2EDuration="1.822470784s" totalImagesPullingTime="423.669173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:21:46.818249524 +0000 UTC m=+1001.765748298" watchObservedRunningTime="2026-09-18 16:21:46.822470784 +0000 UTC m=+1001.769969559"
Sep 18 16:21:46.959948 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:46.959917    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"]
Sep 18 16:21:46.963434 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:21:46.963407    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df144df_c745_4b79_b4f9_404e5d5cb36c.slice/crio-48e3917e6f57822997983a01c9448d0685899444ced5676a61d77bd37b79ae49 WatchSource:0}: Error finding container 48e3917e6f57822997983a01c9448d0685899444ced5676a61d77bd37b79ae49: Status 404 returned error can't find the container with id 48e3917e6f57822997983a01c9448d0685899444ced5676a61d77bd37b79ae49
Sep 18 16:21:47.058513 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:47.058479    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"]
Sep 18 16:21:47.061443 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:21:47.061417    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d37ff3_1220_4f65_a897_1237beeee2e6.slice/crio-b168f0a1a33e129865225a17050666db3e593a4168393a695650463f380b2bb7 WatchSource:0}: Error finding container b168f0a1a33e129865225a17050666db3e593a4168393a695650463f380b2bb7: Status 404 returned error can't find the container with id b168f0a1a33e129865225a17050666db3e593a4168393a695650463f380b2bb7
Sep 18 16:21:47.791117 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:47.791073    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94" event={"ID":"19d37ff3-1220-4f65-a897-1237beeee2e6","Type":"ContainerStarted","Data":"69667bab07eb11e38aadc054b0fec1592b8ac8a1ae1c9a81ce55d383ab256345"}
Sep 18 16:21:47.791598 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:47.791181    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94" event={"ID":"19d37ff3-1220-4f65-a897-1237beeee2e6","Type":"ContainerStarted","Data":"b168f0a1a33e129865225a17050666db3e593a4168393a695650463f380b2bb7"}
Sep 18 16:21:47.791598 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:47.791281    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"
Sep 18 16:21:47.792744 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:47.792717    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj" event={"ID":"4df144df-c745-4b79-b4f9-404e5d5cb36c","Type":"ContainerStarted","Data":"813e4527f6df0ab68865a09bae2e4411afaa85e666f0e6ba0965c5ee587fba23"}
Sep 18 16:21:47.792855 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:47.792759    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj" event={"ID":"4df144df-c745-4b79-b4f9-404e5d5cb36c","Type":"ContainerStarted","Data":"48e3917e6f57822997983a01c9448d0685899444ced5676a61d77bd37b79ae49"}
Sep 18 16:21:47.867921 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:47.867866    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj" podStartSLOduration=1.867847014 podStartE2EDuration="1.867847014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:21:47.866620276 +0000 UTC m=+1002.814119051" watchObservedRunningTime="2026-09-18 16:21:47.867847014 +0000 UTC m=+1002.815345788"
Sep 18 16:21:47.868146 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:47.868088    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94" podStartSLOduration=1.868080984 podStartE2EDuration="1.868080984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:21:47.820726743 +0000 UTC m=+1002.768225528" watchObservedRunningTime="2026-09-18 16:21:47.868080984 +0000 UTC m=+1002.815579798"
Sep 18 16:21:48.798318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:48.798284    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"
Sep 18 16:21:50.552377 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:50.552341    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"]
Sep 18 16:21:50.552874 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:50.552548    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l" podUID="474d3de6-0135-47cb-880b-3fae81cc586a" containerName="maas-api" containerID="cri-o://70a2661f25fc28fff35f1c078755fbcdb9bcd9f384b40ce201d334d7c322adb9" gracePeriod=30
Sep 18 16:21:50.715226 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:50.715188    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"]
Sep 18 16:21:50.715411 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:50.715381    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8" podUID="622a0d89-0bce-4ffa-a580-60e532430592" containerName="payload-processing" containerID="cri-o://270f50a122c7ad4441199b7b3b6ab360c1d3bf00e6bd7673b759aa9a8784a7b5" gracePeriod=30
Sep 18 16:21:50.811323 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:50.811245    2554 generic.go:356] "Generic (PLEG): container finished" podID="474d3de6-0135-47cb-880b-3fae81cc586a" containerID="70a2661f25fc28fff35f1c078755fbcdb9bcd9f384b40ce201d334d7c322adb9" exitCode=0
Sep 18 16:21:50.811447 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:50.811328    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l" event={"ID":"474d3de6-0135-47cb-880b-3fae81cc586a","Type":"ContainerDied","Data":"70a2661f25fc28fff35f1c078755fbcdb9bcd9f384b40ce201d334d7c322adb9"}
Sep 18 16:21:50.845045 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:50.845021    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:21:50.932838 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:50.932801    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4bwc\" (UniqueName: \"kubernetes.io/projected/474d3de6-0135-47cb-880b-3fae81cc586a-kube-api-access-k4bwc\") pod \"474d3de6-0135-47cb-880b-3fae81cc586a\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") "
Sep 18 16:21:50.933009 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:50.932858    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-maas-api-tls\") pod \"474d3de6-0135-47cb-880b-3fae81cc586a\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") "
Sep 18 16:21:50.933009 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:50.932883    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-metrics-certs\") pod \"474d3de6-0135-47cb-880b-3fae81cc586a\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") "
Sep 18 16:21:50.933009 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:50.932910    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/474d3de6-0135-47cb-880b-3fae81cc586a-openshift-service-ca\") pod \"474d3de6-0135-47cb-880b-3fae81cc586a\" (UID: \"474d3de6-0135-47cb-880b-3fae81cc586a\") "
Sep 18 16:21:50.933389 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:50.933358    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474d3de6-0135-47cb-880b-3fae81cc586a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "474d3de6-0135-47cb-880b-3fae81cc586a" (UID: "474d3de6-0135-47cb-880b-3fae81cc586a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:21:50.935942 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:50.935910    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474d3de6-0135-47cb-880b-3fae81cc586a-kube-api-access-k4bwc" (OuterVolumeSpecName: "kube-api-access-k4bwc") pod "474d3de6-0135-47cb-880b-3fae81cc586a" (UID: "474d3de6-0135-47cb-880b-3fae81cc586a"). InnerVolumeSpecName "kube-api-access-k4bwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:21:50.935942 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:50.935915    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "474d3de6-0135-47cb-880b-3fae81cc586a" (UID: "474d3de6-0135-47cb-880b-3fae81cc586a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:21:50.936077 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:50.935927    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "474d3de6-0135-47cb-880b-3fae81cc586a" (UID: "474d3de6-0135-47cb-880b-3fae81cc586a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:21:50.964973 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:50.964949    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"
Sep 18 16:21:51.033938 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.033909    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4bwc\" (UniqueName: \"kubernetes.io/projected/474d3de6-0135-47cb-880b-3fae81cc586a-kube-api-access-k4bwc\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:21:51.033938 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.033935    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-maas-api-tls\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:21:51.033938 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.033944    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/474d3de6-0135-47cb-880b-3fae81cc586a-metrics-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:21:51.034144 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.033954    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/474d3de6-0135-47cb-880b-3fae81cc586a-openshift-service-ca\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:21:51.134613 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.134538    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/622a0d89-0bce-4ffa-a580-60e532430592-plugins-config-volume\") pod \"622a0d89-0bce-4ffa-a580-60e532430592\" (UID: \"622a0d89-0bce-4ffa-a580-60e532430592\") "
Sep 18 16:21:51.134613 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.134601    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hpxw\" (UniqueName: \"kubernetes.io/projected/622a0d89-0bce-4ffa-a580-60e532430592-kube-api-access-4hpxw\") pod \"622a0d89-0bce-4ffa-a580-60e532430592\" (UID: \"622a0d89-0bce-4ffa-a580-60e532430592\") "
Sep 18 16:21:51.134917 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.134889    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/622a0d89-0bce-4ffa-a580-60e532430592-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "622a0d89-0bce-4ffa-a580-60e532430592" (UID: "622a0d89-0bce-4ffa-a580-60e532430592"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:21:51.136786 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.136766    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/622a0d89-0bce-4ffa-a580-60e532430592-kube-api-access-4hpxw" (OuterVolumeSpecName: "kube-api-access-4hpxw") pod "622a0d89-0bce-4ffa-a580-60e532430592" (UID: "622a0d89-0bce-4ffa-a580-60e532430592"). InnerVolumeSpecName "kube-api-access-4hpxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:21:51.235770 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.235733    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/622a0d89-0bce-4ffa-a580-60e532430592-plugins-config-volume\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:21:51.235770 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.235763    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hpxw\" (UniqueName: \"kubernetes.io/projected/622a0d89-0bce-4ffa-a580-60e532430592-kube-api-access-4hpxw\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:21:51.817478 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.817394    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"
Sep 18 16:21:51.817478 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.817437    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l" event={"ID":"474d3de6-0135-47cb-880b-3fae81cc586a","Type":"ContainerDied","Data":"63bc9b47ddabb0eb5029eb1643da213977803559c6c2f988ed3d623ab4b1e8f0"}
Sep 18 16:21:51.817951 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.817499    2554 scope.go:117] "RemoveContainer" containerID="70a2661f25fc28fff35f1c078755fbcdb9bcd9f384b40ce201d334d7c322adb9"
Sep 18 16:21:51.818883 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.818835    2554 generic.go:356] "Generic (PLEG): container finished" podID="622a0d89-0bce-4ffa-a580-60e532430592" containerID="270f50a122c7ad4441199b7b3b6ab360c1d3bf00e6bd7673b759aa9a8784a7b5" exitCode=0
Sep 18 16:21:51.818946 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.818890    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"
Sep 18 16:21:51.818946 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.818914    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8" event={"ID":"622a0d89-0bce-4ffa-a580-60e532430592","Type":"ContainerDied","Data":"270f50a122c7ad4441199b7b3b6ab360c1d3bf00e6bd7673b759aa9a8784a7b5"}
Sep 18 16:21:51.819030 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.818953    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8" event={"ID":"622a0d89-0bce-4ffa-a580-60e532430592","Type":"ContainerDied","Data":"229ef154a46cdbd6df300703e79d854075cf205bbd1016beef7a8e1a9367e9b7"}
Sep 18 16:21:51.826687 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.826670    2554 scope.go:117] "RemoveContainer" containerID="270f50a122c7ad4441199b7b3b6ab360c1d3bf00e6bd7673b759aa9a8784a7b5"
Sep 18 16:21:51.834470 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.834456    2554 scope.go:117] "RemoveContainer" containerID="270f50a122c7ad4441199b7b3b6ab360c1d3bf00e6bd7673b759aa9a8784a7b5"
Sep 18 16:21:51.834772 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:21:51.834751    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270f50a122c7ad4441199b7b3b6ab360c1d3bf00e6bd7673b759aa9a8784a7b5\": container with ID starting with 270f50a122c7ad4441199b7b3b6ab360c1d3bf00e6bd7673b759aa9a8784a7b5 not found: ID does not exist" containerID="270f50a122c7ad4441199b7b3b6ab360c1d3bf00e6bd7673b759aa9a8784a7b5"
Sep 18 16:21:51.834865 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.834778    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270f50a122c7ad4441199b7b3b6ab360c1d3bf00e6bd7673b759aa9a8784a7b5"} err="failed to get container status \"270f50a122c7ad4441199b7b3b6ab360c1d3bf00e6bd7673b759aa9a8784a7b5\": rpc error: code = NotFound desc = could not find container \"270f50a122c7ad4441199b7b3b6ab360c1d3bf00e6bd7673b759aa9a8784a7b5\": container with ID starting with 270f50a122c7ad4441199b7b3b6ab360c1d3bf00e6bd7673b759aa9a8784a7b5 not found: ID does not exist"
Sep 18 16:21:51.849734 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.849714    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"]
Sep 18 16:21:51.852031 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.852011    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-eef18d-79b84c4558-rbg2l"]
Sep 18 16:21:51.890654 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.890621    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"]
Sep 18 16:21:51.905844 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:51.905822    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-eef18d-59f5bf98f5-rdcm8"]
Sep 18 16:21:52.798712 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:52.798680    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:21:53.277377 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:53.277341    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="474d3de6-0135-47cb-880b-3fae81cc586a" path="/var/lib/kubelet/pods/474d3de6-0135-47cb-880b-3fae81cc586a/volumes"
Sep 18 16:21:53.277785 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:53.277734    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="622a0d89-0bce-4ffa-a580-60e532430592" path="/var/lib/kubelet/pods/622a0d89-0bce-4ffa-a580-60e532430592/volumes"
Sep 18 16:21:58.800647 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:58.800617    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"
Sep 18 16:21:59.806393 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:21:59.806363    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"
Sep 18 16:22:13.226476 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.226433    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"]
Sep 18 16:22:13.227007 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.226699    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z" podUID="bcaca9a8-f709-4752-b5eb-42065591f34e" containerName="maas-api" containerID="cri-o://d09daa24a4695dba4739d264f726d2fe9913245ef3e1a75337f8f09e7f7fdbec" gracePeriod=30
Sep 18 16:22:13.350515 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.350467    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"]
Sep 18 16:22:13.351040 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.350959    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94" podUID="19d37ff3-1220-4f65-a897-1237beeee2e6" containerName="payload-processing" containerID="cri-o://69667bab07eb11e38aadc054b0fec1592b8ac8a1ae1c9a81ce55d383ab256345" gracePeriod=30
Sep 18 16:22:13.376858 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.376823    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"]
Sep 18 16:22:13.377047 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.377025    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj" podUID="4df144df-c745-4b79-b4f9-404e5d5cb36c" containerName="payload-pre-processing" containerID="cri-o://813e4527f6df0ab68865a09bae2e4411afaa85e666f0e6ba0965c5ee587fba23" gracePeriod=30
Sep 18 16:22:13.503893 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.503863    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:22:13.536024 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.535987    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcfgd\" (UniqueName: \"kubernetes.io/projected/bcaca9a8-f709-4752-b5eb-42065591f34e-kube-api-access-zcfgd\") pod \"bcaca9a8-f709-4752-b5eb-42065591f34e\" (UID: \"bcaca9a8-f709-4752-b5eb-42065591f34e\") "
Sep 18 16:22:13.536024 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.536028    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcaca9a8-f709-4752-b5eb-42065591f34e-metrics-certs\") pod \"bcaca9a8-f709-4752-b5eb-42065591f34e\" (UID: \"bcaca9a8-f709-4752-b5eb-42065591f34e\") "
Sep 18 16:22:13.536271 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.536055    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcaca9a8-f709-4752-b5eb-42065591f34e-openshift-service-ca\") pod \"bcaca9a8-f709-4752-b5eb-42065591f34e\" (UID: \"bcaca9a8-f709-4752-b5eb-42065591f34e\") "
Sep 18 16:22:13.536271 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.536112    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcaca9a8-f709-4752-b5eb-42065591f34e-maas-api-tls\") pod \"bcaca9a8-f709-4752-b5eb-42065591f34e\" (UID: \"bcaca9a8-f709-4752-b5eb-42065591f34e\") "
Sep 18 16:22:13.539542 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.537408    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcaca9a8-f709-4752-b5eb-42065591f34e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bcaca9a8-f709-4752-b5eb-42065591f34e" (UID: "bcaca9a8-f709-4752-b5eb-42065591f34e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:22:13.539542 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.539113    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcaca9a8-f709-4752-b5eb-42065591f34e-kube-api-access-zcfgd" (OuterVolumeSpecName: "kube-api-access-zcfgd") pod "bcaca9a8-f709-4752-b5eb-42065591f34e" (UID: "bcaca9a8-f709-4752-b5eb-42065591f34e"). InnerVolumeSpecName "kube-api-access-zcfgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:22:13.539542 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.539426    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcaca9a8-f709-4752-b5eb-42065591f34e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bcaca9a8-f709-4752-b5eb-42065591f34e" (UID: "bcaca9a8-f709-4752-b5eb-42065591f34e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:22:13.541306 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.541206    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcaca9a8-f709-4752-b5eb-42065591f34e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bcaca9a8-f709-4752-b5eb-42065591f34e" (UID: "bcaca9a8-f709-4752-b5eb-42065591f34e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:22:13.608638 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.608612    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"
Sep 18 16:22:13.637301 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.637277    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d37ff3-1220-4f65-a897-1237beeee2e6-plugins-config-volume\") pod \"19d37ff3-1220-4f65-a897-1237beeee2e6\" (UID: \"19d37ff3-1220-4f65-a897-1237beeee2e6\") "
Sep 18 16:22:13.637410 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.637346    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8qjh\" (UniqueName: \"kubernetes.io/projected/19d37ff3-1220-4f65-a897-1237beeee2e6-kube-api-access-c8qjh\") pod \"19d37ff3-1220-4f65-a897-1237beeee2e6\" (UID: \"19d37ff3-1220-4f65-a897-1237beeee2e6\") "
Sep 18 16:22:13.637680 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.637653    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d37ff3-1220-4f65-a897-1237beeee2e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19d37ff3-1220-4f65-a897-1237beeee2e6" (UID: "19d37ff3-1220-4f65-a897-1237beeee2e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:22:13.637796 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.637680    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcfgd\" (UniqueName: \"kubernetes.io/projected/bcaca9a8-f709-4752-b5eb-42065591f34e-kube-api-access-zcfgd\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:22:13.637796 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.637700    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcaca9a8-f709-4752-b5eb-42065591f34e-metrics-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:22:13.637796 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.637712    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcaca9a8-f709-4752-b5eb-42065591f34e-openshift-service-ca\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:22:13.637796 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.637721    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcaca9a8-f709-4752-b5eb-42065591f34e-maas-api-tls\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:22:13.639958 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.639934    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d37ff3-1220-4f65-a897-1237beeee2e6-kube-api-access-c8qjh" (OuterVolumeSpecName: "kube-api-access-c8qjh") pod "19d37ff3-1220-4f65-a897-1237beeee2e6" (UID: "19d37ff3-1220-4f65-a897-1237beeee2e6"). InnerVolumeSpecName "kube-api-access-c8qjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:22:13.645651 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.645629    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"
Sep 18 16:22:13.738371 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.738284    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4df144df-c745-4b79-b4f9-404e5d5cb36c-plugins-config-volume\") pod \"4df144df-c745-4b79-b4f9-404e5d5cb36c\" (UID: \"4df144df-c745-4b79-b4f9-404e5d5cb36c\") "
Sep 18 16:22:13.738371 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.738327    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nppjr\" (UniqueName: \"kubernetes.io/projected/4df144df-c745-4b79-b4f9-404e5d5cb36c-kube-api-access-nppjr\") pod \"4df144df-c745-4b79-b4f9-404e5d5cb36c\" (UID: \"4df144df-c745-4b79-b4f9-404e5d5cb36c\") "
Sep 18 16:22:13.738669 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.738645    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df144df-c745-4b79-b4f9-404e5d5cb36c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4df144df-c745-4b79-b4f9-404e5d5cb36c" (UID: "4df144df-c745-4b79-b4f9-404e5d5cb36c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:22:13.738669 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.738666    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d37ff3-1220-4f65-a897-1237beeee2e6-plugins-config-volume\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:22:13.738783 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.738678    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8qjh\" (UniqueName: \"kubernetes.io/projected/19d37ff3-1220-4f65-a897-1237beeee2e6-kube-api-access-c8qjh\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:22:13.740581 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.740559    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df144df-c745-4b79-b4f9-404e5d5cb36c-kube-api-access-nppjr" (OuterVolumeSpecName: "kube-api-access-nppjr") pod "4df144df-c745-4b79-b4f9-404e5d5cb36c" (UID: "4df144df-c745-4b79-b4f9-404e5d5cb36c"). InnerVolumeSpecName "kube-api-access-nppjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:22:13.840063 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.840029    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4df144df-c745-4b79-b4f9-404e5d5cb36c-plugins-config-volume\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:22:13.840063 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.840057    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nppjr\" (UniqueName: \"kubernetes.io/projected/4df144df-c745-4b79-b4f9-404e5d5cb36c-kube-api-access-nppjr\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:22:13.937984 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.937945    2554 generic.go:356] "Generic (PLEG): container finished" podID="4df144df-c745-4b79-b4f9-404e5d5cb36c" containerID="813e4527f6df0ab68865a09bae2e4411afaa85e666f0e6ba0965c5ee587fba23" exitCode=0
Sep 18 16:22:13.938175 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.937992    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"
Sep 18 16:22:13.938175 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.938034    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj" event={"ID":"4df144df-c745-4b79-b4f9-404e5d5cb36c","Type":"ContainerDied","Data":"813e4527f6df0ab68865a09bae2e4411afaa85e666f0e6ba0965c5ee587fba23"}
Sep 18 16:22:13.938175 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.938087    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj" event={"ID":"4df144df-c745-4b79-b4f9-404e5d5cb36c","Type":"ContainerDied","Data":"48e3917e6f57822997983a01c9448d0685899444ced5676a61d77bd37b79ae49"}
Sep 18 16:22:13.938175 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.938115    2554 scope.go:117] "RemoveContainer" containerID="813e4527f6df0ab68865a09bae2e4411afaa85e666f0e6ba0965c5ee587fba23"
Sep 18 16:22:13.939436 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.939404    2554 generic.go:356] "Generic (PLEG): container finished" podID="bcaca9a8-f709-4752-b5eb-42065591f34e" containerID="d09daa24a4695dba4739d264f726d2fe9913245ef3e1a75337f8f09e7f7fdbec" exitCode=0
Sep 18 16:22:13.939436 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.939428    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z" event={"ID":"bcaca9a8-f709-4752-b5eb-42065591f34e","Type":"ContainerDied","Data":"d09daa24a4695dba4739d264f726d2fe9913245ef3e1a75337f8f09e7f7fdbec"}
Sep 18 16:22:13.939797 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.939449    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"
Sep 18 16:22:13.939797 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.939474    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z" event={"ID":"bcaca9a8-f709-4752-b5eb-42065591f34e","Type":"ContainerDied","Data":"ad9829816e9b903c8b77e534501c923277ff6ff8e1ef20799cc4fdc9754348fe"}
Sep 18 16:22:13.940981 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.940958    2554 generic.go:356] "Generic (PLEG): container finished" podID="19d37ff3-1220-4f65-a897-1237beeee2e6" containerID="69667bab07eb11e38aadc054b0fec1592b8ac8a1ae1c9a81ce55d383ab256345" exitCode=0
Sep 18 16:22:13.941083 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.940991    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"
Sep 18 16:22:13.941083 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.940990    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94" event={"ID":"19d37ff3-1220-4f65-a897-1237beeee2e6","Type":"ContainerDied","Data":"69667bab07eb11e38aadc054b0fec1592b8ac8a1ae1c9a81ce55d383ab256345"}
Sep 18 16:22:13.941197 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.941095    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94" event={"ID":"19d37ff3-1220-4f65-a897-1237beeee2e6","Type":"ContainerDied","Data":"b168f0a1a33e129865225a17050666db3e593a4168393a695650463f380b2bb7"}
Sep 18 16:22:13.947312 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.947292    2554 scope.go:117] "RemoveContainer" containerID="813e4527f6df0ab68865a09bae2e4411afaa85e666f0e6ba0965c5ee587fba23"
Sep 18 16:22:13.947595 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:22:13.947573    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"813e4527f6df0ab68865a09bae2e4411afaa85e666f0e6ba0965c5ee587fba23\": container with ID starting with 813e4527f6df0ab68865a09bae2e4411afaa85e666f0e6ba0965c5ee587fba23 not found: ID does not exist" containerID="813e4527f6df0ab68865a09bae2e4411afaa85e666f0e6ba0965c5ee587fba23"
Sep 18 16:22:13.947673 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.947601    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813e4527f6df0ab68865a09bae2e4411afaa85e666f0e6ba0965c5ee587fba23"} err="failed to get container status \"813e4527f6df0ab68865a09bae2e4411afaa85e666f0e6ba0965c5ee587fba23\": rpc error: code = NotFound desc = could not find container \"813e4527f6df0ab68865a09bae2e4411afaa85e666f0e6ba0965c5ee587fba23\": container with ID starting with 813e4527f6df0ab68865a09bae2e4411afaa85e666f0e6ba0965c5ee587fba23 not found: ID does not exist"
Sep 18 16:22:13.947673 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.947617    2554 scope.go:117] "RemoveContainer" containerID="d09daa24a4695dba4739d264f726d2fe9913245ef3e1a75337f8f09e7f7fdbec"
Sep 18 16:22:13.957395 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.957379    2554 scope.go:117] "RemoveContainer" containerID="d09daa24a4695dba4739d264f726d2fe9913245ef3e1a75337f8f09e7f7fdbec"
Sep 18 16:22:13.957690 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:22:13.957672    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d09daa24a4695dba4739d264f726d2fe9913245ef3e1a75337f8f09e7f7fdbec\": container with ID starting with d09daa24a4695dba4739d264f726d2fe9913245ef3e1a75337f8f09e7f7fdbec not found: ID does not exist" containerID="d09daa24a4695dba4739d264f726d2fe9913245ef3e1a75337f8f09e7f7fdbec"
Sep 18 16:22:13.957748 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.957698    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09daa24a4695dba4739d264f726d2fe9913245ef3e1a75337f8f09e7f7fdbec"} err="failed to get container status \"d09daa24a4695dba4739d264f726d2fe9913245ef3e1a75337f8f09e7f7fdbec\": rpc error: code = NotFound desc = could not find container \"d09daa24a4695dba4739d264f726d2fe9913245ef3e1a75337f8f09e7f7fdbec\": container with ID starting with d09daa24a4695dba4739d264f726d2fe9913245ef3e1a75337f8f09e7f7fdbec not found: ID does not exist"
Sep 18 16:22:13.957748 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.957714    2554 scope.go:117] "RemoveContainer" containerID="69667bab07eb11e38aadc054b0fec1592b8ac8a1ae1c9a81ce55d383ab256345"
Sep 18 16:22:13.965380 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.965364    2554 scope.go:117] "RemoveContainer" containerID="69667bab07eb11e38aadc054b0fec1592b8ac8a1ae1c9a81ce55d383ab256345"
Sep 18 16:22:13.965676 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:22:13.965659    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69667bab07eb11e38aadc054b0fec1592b8ac8a1ae1c9a81ce55d383ab256345\": container with ID starting with 69667bab07eb11e38aadc054b0fec1592b8ac8a1ae1c9a81ce55d383ab256345 not found: ID does not exist" containerID="69667bab07eb11e38aadc054b0fec1592b8ac8a1ae1c9a81ce55d383ab256345"
Sep 18 16:22:13.965734 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:13.965682    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69667bab07eb11e38aadc054b0fec1592b8ac8a1ae1c9a81ce55d383ab256345"} err="failed to get container status \"69667bab07eb11e38aadc054b0fec1592b8ac8a1ae1c9a81ce55d383ab256345\": rpc error: code = NotFound desc = could not find container \"69667bab07eb11e38aadc054b0fec1592b8ac8a1ae1c9a81ce55d383ab256345\": container with ID starting with 69667bab07eb11e38aadc054b0fec1592b8ac8a1ae1c9a81ce55d383ab256345 not found: ID does not exist"
Sep 18 16:22:14.215767 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:14.215493    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"]
Sep 18 16:22:14.217415 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:14.217394    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-11860c1b-86f8cc9454-n8psj"]
Sep 18 16:22:14.236082 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:14.236056    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"]
Sep 18 16:22:14.249592 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:14.249567    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-11860c1b-64798c5db9-9wn94"]
Sep 18 16:22:14.271079 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:14.271053    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"]
Sep 18 16:22:14.272913 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:14.272888    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-11860c1b-7f496d5895-z8n7z"]
Sep 18 16:22:15.278009 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:15.277975    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19d37ff3-1220-4f65-a897-1237beeee2e6" path="/var/lib/kubelet/pods/19d37ff3-1220-4f65-a897-1237beeee2e6/volumes"
Sep 18 16:22:15.278389 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:15.278350    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4df144df-c745-4b79-b4f9-404e5d5cb36c" path="/var/lib/kubelet/pods/4df144df-c745-4b79-b4f9-404e5d5cb36c/volumes"
Sep 18 16:22:15.278734 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:15.278721    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcaca9a8-f709-4752-b5eb-42065591f34e" path="/var/lib/kubelet/pods/bcaca9a8-f709-4752-b5eb-42065591f34e/volumes"
Sep 18 16:22:28.680601 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.680567    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"]
Sep 18 16:22:28.681151 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.681131    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df144df-c745-4b79-b4f9-404e5d5cb36c" containerName="payload-pre-processing"
Sep 18 16:22:28.681241 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.681155    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df144df-c745-4b79-b4f9-404e5d5cb36c" containerName="payload-pre-processing"
Sep 18 16:22:28.681241 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.681175    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19d37ff3-1220-4f65-a897-1237beeee2e6" containerName="payload-processing"
Sep 18 16:22:28.681241 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.681184    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="19d37ff3-1220-4f65-a897-1237beeee2e6" containerName="payload-processing"
Sep 18 16:22:28.681241 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.681199    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcaca9a8-f709-4752-b5eb-42065591f34e" containerName="maas-api"
Sep 18 16:22:28.681241 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.681207    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcaca9a8-f709-4752-b5eb-42065591f34e" containerName="maas-api"
Sep 18 16:22:28.681241 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.681220    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="474d3de6-0135-47cb-880b-3fae81cc586a" containerName="maas-api"
Sep 18 16:22:28.681241 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.681228    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="474d3de6-0135-47cb-880b-3fae81cc586a" containerName="maas-api"
Sep 18 16:22:28.681608 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.681250    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="622a0d89-0bce-4ffa-a580-60e532430592" containerName="payload-processing"
Sep 18 16:22:28.681608 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.681259    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="622a0d89-0bce-4ffa-a580-60e532430592" containerName="payload-processing"
Sep 18 16:22:28.681608 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.681363    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="622a0d89-0bce-4ffa-a580-60e532430592" containerName="payload-processing"
Sep 18 16:22:28.681608 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.681377    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="474d3de6-0135-47cb-880b-3fae81cc586a" containerName="maas-api"
Sep 18 16:22:28.681608 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.681384    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df144df-c745-4b79-b4f9-404e5d5cb36c" containerName="payload-pre-processing"
Sep 18 16:22:28.681608 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.681396    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcaca9a8-f709-4752-b5eb-42065591f34e" containerName="maas-api"
Sep 18 16:22:28.681608 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.681405    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="19d37ff3-1220-4f65-a897-1237beeee2e6" containerName="payload-processing"
Sep 18 16:22:28.684539 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.684500    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.688859 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.688832    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-b73e26-openshift-default-dockercfg-4whr9\""
Sep 18 16:22:28.689456 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.689439    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 16:22:28.703687 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.703663    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"]
Sep 18 16:22:28.777818 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.777789    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-workload-socket\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.777937 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.777848    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-envoy\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.777937 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.777890    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-credential-socket\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.778018 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.777943    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-workload-certs\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.778018 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.777970    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-token\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.778018 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.777991    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h9cb\" (UniqueName: \"kubernetes.io/projected/a7d69023-7ac5-432b-982c-f4ef453e0103-kube-api-access-2h9cb\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.778018 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.778013    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-data\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.778169 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.778039    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d69023-7ac5-432b-982c-f4ef453e0103-istiod-ca-cert\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.778169 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.778059    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-podinfo\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.878831 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.878795    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-envoy\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.878990 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.878844    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-credential-socket\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.878990 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.878873    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-workload-certs\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.878990 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.878900    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-token\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.878990 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.878929    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h9cb\" (UniqueName: \"kubernetes.io/projected/a7d69023-7ac5-432b-982c-f4ef453e0103-kube-api-access-2h9cb\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.878990 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.878961    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-data\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.879254 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.878997    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d69023-7ac5-432b-982c-f4ef453e0103-istiod-ca-cert\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.879254 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.879022    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-podinfo\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.879254 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.879093    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-workload-socket\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.879460 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.879283    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-credential-socket\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.879460 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.879318    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-workload-certs\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.879847 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.879768    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d69023-7ac5-432b-982c-f4ef453e0103-istiod-ca-cert\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.879847 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.879783    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-workload-socket\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.879847 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.879833    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-data\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.881273 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.881253    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-envoy\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.881550 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.881513    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-podinfo\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.895624 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.895601    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h9cb\" (UniqueName: \"kubernetes.io/projected/a7d69023-7ac5-432b-982c-f4ef453e0103-kube-api-access-2h9cb\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.899692 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.899669    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-token\") pod \"e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") " pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:28.994999 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:28.994920    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:29.284333 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:29.284254    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"]
Sep 18 16:22:29.288961 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:22:29.288935    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d69023_7ac5_432b_982c_f4ef453e0103.slice/crio-71ceb4cfbe7d3a4182a76da1fe5c4f79a462eb0ae52f8136103e9b9c226b8476 WatchSource:0}: Error finding container 71ceb4cfbe7d3a4182a76da1fe5c4f79a462eb0ae52f8136103e9b9c226b8476: Status 404 returned error can't find the container with id 71ceb4cfbe7d3a4182a76da1fe5c4f79a462eb0ae52f8136103e9b9c226b8476
Sep 18 16:22:30.025443 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:30.025404    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q" event={"ID":"a7d69023-7ac5-432b-982c-f4ef453e0103","Type":"ContainerStarted","Data":"71ceb4cfbe7d3a4182a76da1fe5c4f79a462eb0ae52f8136103e9b9c226b8476"}
Sep 18 16:22:32.143354 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:32.143313    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:22:32.143637 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:32.143393    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:22:32.143637 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:32.143425    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:22:33.044323 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:33.044289    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q" event={"ID":"a7d69023-7ac5-432b-982c-f4ef453e0103","Type":"ContainerStarted","Data":"11f6f621de22e1e39c8bb94d65b8c274696c27bdd3dab6bad61c1d1a7d9bb3fe"}
Sep 18 16:22:33.084576 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:33.084486    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q" podStartSLOduration=2.232451517 podStartE2EDuration="5.084463955s" totalImagesPullingTime="2.852012438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:33.076073085 +0000 UTC m=+1048.023571864" watchObservedRunningTime="2026-09-18 16:22:33.084463955 +0000 UTC m=+1048.031962732"
Sep 18 16:22:33.995851 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:33.995818    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:34.001177 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:34.001150    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:34.050800 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:34.050754    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:34.051804 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:34.051784    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:22:41.393845 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:41.393808    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"]
Sep 18 16:22:41.397697 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:41.397677    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:41.401301 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:41.401280    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-b73e26-serving-cert\""
Sep 18 16:22:41.430808 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:41.430775    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"]
Sep 18 16:22:41.595509 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:41.595476    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cvjn\" (UniqueName: \"kubernetes.io/projected/71e50642-38b5-480d-bd85-f6846cade66f-kube-api-access-8cvjn\") pod \"maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:41.595694 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:41.595589    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e50642-38b5-480d-bd85-f6846cade66f-metrics-certs\") pod \"maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:41.595694 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:41.595637    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e50642-38b5-480d-bd85-f6846cade66f-openshift-service-ca\") pod \"maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:41.595694 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:41.595663    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e50642-38b5-480d-bd85-f6846cade66f-maas-api-tls\") pod \"maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:41.696375 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:41.696341    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cvjn\" (UniqueName: \"kubernetes.io/projected/71e50642-38b5-480d-bd85-f6846cade66f-kube-api-access-8cvjn\") pod \"maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:41.696588 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:41.696401    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e50642-38b5-480d-bd85-f6846cade66f-metrics-certs\") pod \"maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:41.696588 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:41.696426    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e50642-38b5-480d-bd85-f6846cade66f-openshift-service-ca\") pod \"maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:41.696588 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:41.696442    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e50642-38b5-480d-bd85-f6846cade66f-maas-api-tls\") pod \"maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:41.696588 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:22:41.696584    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-serving-cert: secret "maas-api-e2e-explicit-b73e26-serving-cert" not found
Sep 18 16:22:41.696851 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:22:41.696645    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71e50642-38b5-480d-bd85-f6846cade66f-maas-api-tls podName:71e50642-38b5-480d-bd85-f6846cade66f nodeName:}" failed. No retries permitted until 2026-09-18 16:22:42.19662715 +0000 UTC m=+1057.144125903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71e50642-38b5-480d-bd85-f6846cade66f-maas-api-tls") pod "maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8" (UID: "71e50642-38b5-480d-bd85-f6846cade66f") : secret "maas-api-e2e-explicit-b73e26-serving-cert" not found
Sep 18 16:22:41.697097 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:41.697078    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e50642-38b5-480d-bd85-f6846cade66f-openshift-service-ca\") pod \"maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:41.699135 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:41.699112    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e50642-38b5-480d-bd85-f6846cade66f-metrics-certs\") pod \"maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:41.715429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:41.715398    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cvjn\" (UniqueName: \"kubernetes.io/projected/71e50642-38b5-480d-bd85-f6846cade66f-kube-api-access-8cvjn\") pod \"maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:42.200010 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.199969    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e50642-38b5-480d-bd85-f6846cade66f-maas-api-tls\") pod \"maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:42.202508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.202476    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e50642-38b5-480d-bd85-f6846cade66f-maas-api-tls\") pod \"maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:42.307838 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.307804    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:42.619753 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.619724    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"]
Sep 18 16:22:42.622084 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:22:42.622058    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e50642_38b5_480d_bd85_f6846cade66f.slice/crio-5f476972d99cba24d71b7dec8a06200e66e77a445334e0e1c8296e00761d8378 WatchSource:0}: Error finding container 5f476972d99cba24d71b7dec8a06200e66e77a445334e0e1c8296e00761d8378: Status 404 returned error can't find the container with id 5f476972d99cba24d71b7dec8a06200e66e77a445334e0e1c8296e00761d8378
Sep 18 16:22:42.630213 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.630188    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"]
Sep 18 16:22:42.634811 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.634791    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"
Sep 18 16:22:42.639883 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.639865    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b73e26-dockercfg-d4txx\""
Sep 18 16:22:42.640361 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.640344    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b73e26\""
Sep 18 16:22:42.659757 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.659734    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"]
Sep 18 16:22:42.687442 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.687420    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"]
Sep 18 16:22:42.690888 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.690874    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"
Sep 18 16:22:42.727544 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.727503    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"]
Sep 18 16:22:42.806828 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.806794    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpsgh\" (UniqueName: \"kubernetes.io/projected/0d9615e5-ecaf-48dd-a8fb-ce1fd420b101-kube-api-access-jpsgh\") pod \"payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm\" (UID: \"0d9615e5-ecaf-48dd-a8fb-ce1fd420b101\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"
Sep 18 16:22:42.806996 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.806870    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d8a731-67fd-416f-a0b4-449614660fbe-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx\" (UID: \"33d8a731-67fd-416f-a0b4-449614660fbe\") " pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"
Sep 18 16:22:42.806996 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.806893    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d9615e5-ecaf-48dd-a8fb-ce1fd420b101-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm\" (UID: \"0d9615e5-ecaf-48dd-a8fb-ce1fd420b101\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"
Sep 18 16:22:42.806996 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.806972    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxkrh\" (UniqueName: \"kubernetes.io/projected/33d8a731-67fd-416f-a0b4-449614660fbe-kube-api-access-sxkrh\") pod \"payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx\" (UID: \"33d8a731-67fd-416f-a0b4-449614660fbe\") " pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"
Sep 18 16:22:42.908247 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.908159    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d8a731-67fd-416f-a0b4-449614660fbe-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx\" (UID: \"33d8a731-67fd-416f-a0b4-449614660fbe\") " pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"
Sep 18 16:22:42.908247 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.908193    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d9615e5-ecaf-48dd-a8fb-ce1fd420b101-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm\" (UID: \"0d9615e5-ecaf-48dd-a8fb-ce1fd420b101\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"
Sep 18 16:22:42.908442 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.908272    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxkrh\" (UniqueName: \"kubernetes.io/projected/33d8a731-67fd-416f-a0b4-449614660fbe-kube-api-access-sxkrh\") pod \"payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx\" (UID: \"33d8a731-67fd-416f-a0b4-449614660fbe\") " pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"
Sep 18 16:22:42.908442 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.908326    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpsgh\" (UniqueName: \"kubernetes.io/projected/0d9615e5-ecaf-48dd-a8fb-ce1fd420b101-kube-api-access-jpsgh\") pod \"payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm\" (UID: \"0d9615e5-ecaf-48dd-a8fb-ce1fd420b101\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"
Sep 18 16:22:42.908969 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.908951    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d8a731-67fd-416f-a0b4-449614660fbe-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx\" (UID: \"33d8a731-67fd-416f-a0b4-449614660fbe\") " pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"
Sep 18 16:22:42.909040 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.909022    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d9615e5-ecaf-48dd-a8fb-ce1fd420b101-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm\" (UID: \"0d9615e5-ecaf-48dd-a8fb-ce1fd420b101\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"
Sep 18 16:22:42.925516 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.925488    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpsgh\" (UniqueName: \"kubernetes.io/projected/0d9615e5-ecaf-48dd-a8fb-ce1fd420b101-kube-api-access-jpsgh\") pod \"payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm\" (UID: \"0d9615e5-ecaf-48dd-a8fb-ce1fd420b101\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"
Sep 18 16:22:42.931626 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.931605    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxkrh\" (UniqueName: \"kubernetes.io/projected/33d8a731-67fd-416f-a0b4-449614660fbe-kube-api-access-sxkrh\") pod \"payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx\" (UID: \"33d8a731-67fd-416f-a0b4-449614660fbe\") " pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"
Sep 18 16:22:42.945442 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:42.945415    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"
Sep 18 16:22:43.000949 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:43.000918    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"
Sep 18 16:22:43.101140 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:43.101104    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8" event={"ID":"71e50642-38b5-480d-bd85-f6846cade66f","Type":"ContainerStarted","Data":"5f476972d99cba24d71b7dec8a06200e66e77a445334e0e1c8296e00761d8378"}
Sep 18 16:22:43.233778 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:43.233750    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"]
Sep 18 16:22:43.236242 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:22:43.236213    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d8a731_67fd_416f_a0b4_449614660fbe.slice/crio-76034f7e01b4e86d1e11f2e88f6cc62c08eb321af6f4c706841596e5d1d482b1 WatchSource:0}: Error finding container 76034f7e01b4e86d1e11f2e88f6cc62c08eb321af6f4c706841596e5d1d482b1: Status 404 returned error can't find the container with id 76034f7e01b4e86d1e11f2e88f6cc62c08eb321af6f4c706841596e5d1d482b1
Sep 18 16:22:43.295151 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:43.295123    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"]
Sep 18 16:22:43.295313 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:22:43.295292    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9615e5_ecaf_48dd_a8fb_ce1fd420b101.slice/crio-d688191a50f0499e53d63d20b7922a1c70da98d93ca86102a26faf61f4545a14 WatchSource:0}: Error finding container d688191a50f0499e53d63d20b7922a1c70da98d93ca86102a26faf61f4545a14: Status 404 returned error can't find the container with id d688191a50f0499e53d63d20b7922a1c70da98d93ca86102a26faf61f4545a14
Sep 18 16:22:44.109903 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:44.109862    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx" event={"ID":"33d8a731-67fd-416f-a0b4-449614660fbe","Type":"ContainerStarted","Data":"871232758cc6f442edf7f6950b6d91ff124a77c64a7037ace4f79e88999c37d8"}
Sep 18 16:22:44.110391 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:44.109985    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"
Sep 18 16:22:44.110391 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:44.110018    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx" event={"ID":"33d8a731-67fd-416f-a0b4-449614660fbe","Type":"ContainerStarted","Data":"76034f7e01b4e86d1e11f2e88f6cc62c08eb321af6f4c706841596e5d1d482b1"}
Sep 18 16:22:44.111534 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:44.111501    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm" event={"ID":"0d9615e5-ecaf-48dd-a8fb-ce1fd420b101","Type":"ContainerStarted","Data":"923a7aff4bc56e67ffb78dd9d803e2b0e51cc9f5c74875d54caa8b77df4dd118"}
Sep 18 16:22:44.111650 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:44.111553    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm" event={"ID":"0d9615e5-ecaf-48dd-a8fb-ce1fd420b101","Type":"ContainerStarted","Data":"d688191a50f0499e53d63d20b7922a1c70da98d93ca86102a26faf61f4545a14"}
Sep 18 16:22:44.111650 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:44.111616    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"
Sep 18 16:22:44.113087 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:44.113065    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8" event={"ID":"71e50642-38b5-480d-bd85-f6846cade66f","Type":"ContainerStarted","Data":"148992677b7088bbf10ced978a49489c1c966da958441ce1a9cbe9adabe16bc5"}
Sep 18 16:22:44.113196 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:44.113182    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:44.133364 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:44.133323    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx" podStartSLOduration=2.133310191 podStartE2EDuration="2.133310191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:44.130631265 +0000 UTC m=+1059.078130040" watchObservedRunningTime="2026-09-18 16:22:44.133310191 +0000 UTC m=+1059.080808962"
Sep 18 16:22:44.155092 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:44.155058    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm" podStartSLOduration=2.15504679 podStartE2EDuration="2.15504679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:44.152922575 +0000 UTC m=+1059.100421346" watchObservedRunningTime="2026-09-18 16:22:44.15504679 +0000 UTC m=+1059.102545565"
Sep 18 16:22:44.179135 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:44.179097    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8" podStartSLOduration=2.684034274 podStartE2EDuration="3.179084368s" totalImagesPullingTime="495.050094ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:44.178003547 +0000 UTC m=+1059.125502323" watchObservedRunningTime="2026-09-18 16:22:44.179084368 +0000 UTC m=+1059.126583142"
Sep 18 16:22:50.123842 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:50.123761    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:22:55.121424 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:55.121391    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"
Sep 18 16:22:55.122030 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:55.121475    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"
Sep 18 16:22:59.210867 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:59.210836    2554 generic.go:356] "Generic (PLEG): container finished" podID="a7d69023-7ac5-432b-982c-f4ef453e0103" containerID="11f6f621de22e1e39c8bb94d65b8c274696c27bdd3dab6bad61c1d1a7d9bb3fe" exitCode=0
Sep 18 16:22:59.211273 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:59.210911    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q" event={"ID":"a7d69023-7ac5-432b-982c-f4ef453e0103","Type":"ContainerDied","Data":"11f6f621de22e1e39c8bb94d65b8c274696c27bdd3dab6bad61c1d1a7d9bb3fe"}
Sep 18 16:22:59.211336 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:59.211320    2554 scope.go:117] "RemoveContainer" containerID="11f6f621de22e1e39c8bb94d65b8c274696c27bdd3dab6bad61c1d1a7d9bb3fe"
Sep 18 16:22:59.212538 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:59.212490    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:22:59.212603 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:59.212578    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:22:59.212644 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:22:59.212613    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:23:00.218355 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:00.218324    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q" event={"ID":"a7d69023-7ac5-432b-982c-f4ef453e0103","Type":"ContainerStarted","Data":"41a10eee61abaa2e5aed1d412ee46d3fb42e7c0bec6163d9e9d0e44fc5036135"}
Sep 18 16:23:00.995764 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:00.995726    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:23:00.997044 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:00.997018    2554 patch_prober.go:28] interesting pod/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 18 16:23:00.997201 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:00.997080    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q" podUID="a7d69023-7ac5-432b-982c-f4ef453e0103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 18 16:23:01.996331 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:01.996291    2554 patch_prober.go:28] interesting pod/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 18 16:23:01.996853 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:01.996360    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q" podUID="a7d69023-7ac5-432b-982c-f4ef453e0103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 18 16:23:02.995787 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:02.995751    2554 patch_prober.go:28] interesting pod/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 18 16:23:02.995973 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:02.995809    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q" podUID="a7d69023-7ac5-432b-982c-f4ef453e0103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 18 16:23:03.996080 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:03.996048    2554 patch_prober.go:28] interesting pod/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 18 16:23:03.996475 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:03.996104    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q" podUID="a7d69023-7ac5-432b-982c-f4ef453e0103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 18 16:23:04.052589 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:04.051688    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:23:04.996224 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:04.996184    2554 patch_prober.go:28] interesting pod/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 18 16:23:04.996714 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:04.996250    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q" podUID="a7d69023-7ac5-432b-982c-f4ef453e0103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 18 16:23:06.000364 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:06.000327    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:23:06.001381 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:06.001363    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:23:51.576714 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.576672    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"]
Sep 18 16:23:51.579330 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.579308    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.587365 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.587342    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b73e26\"/\"openshift-service-ca.crt\""
Sep 18 16:23:51.587474 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.587409    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b73e26\"/\"default-dockercfg-lj2dt\""
Sep 18 16:23:51.587474 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.587420    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b73e26\"/\"explicit-model-b73e26-kserve-self-signed-certs\""
Sep 18 16:23:51.587474 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.587459    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b73e26\"/\"kube-root-ca.crt\""
Sep 18 16:23:51.602037 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.602017    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"]
Sep 18 16:23:51.661196 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.661167    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-home\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.661334 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.661255    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-model-cache\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.661334 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.661311    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-tmp-dir\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.661460 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.661363    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-dshm\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.661460 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.661399    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf6bg\" (UniqueName: \"kubernetes.io/projected/4e284191-4656-450d-8bb2-b768ba5a107b-kube-api-access-bf6bg\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.661460 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.661450    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e284191-4656-450d-8bb2-b768ba5a107b-tls-certs\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.762346 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.762320    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e284191-4656-450d-8bb2-b768ba5a107b-tls-certs\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.762464 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.762361    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-home\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.762464 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.762410    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-model-cache\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.762623 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.762510    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-tmp-dir\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.762623 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.762610    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-dshm\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.762739 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.762631    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf6bg\" (UniqueName: \"kubernetes.io/projected/4e284191-4656-450d-8bb2-b768ba5a107b-kube-api-access-bf6bg\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.762795 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.762763    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-model-cache\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.762848 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.762802    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-home\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.762917 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.762898    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-tmp-dir\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.764810 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.764791    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-dshm\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.765130 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.765110    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e284191-4656-450d-8bb2-b768ba5a107b-tls-certs\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.776223 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.776195    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf6bg\" (UniqueName: \"kubernetes.io/projected/4e284191-4656-450d-8bb2-b768ba5a107b-kube-api-access-bf6bg\") pod \"explicit-model-b73e26-kserve-c88b59747-5nm4c\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") " pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:51.888307 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:51.888224    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:23:52.178711 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:52.178689    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"]
Sep 18 16:23:52.181493 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:23:52.181454    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e284191_4656_450d_8bb2_b768ba5a107b.slice/crio-47145709fb37fa132e8d38fa4cf45749b4bb387bd9e0d221dd0f574e8724cc81 WatchSource:0}: Error finding container 47145709fb37fa132e8d38fa4cf45749b4bb387bd9e0d221dd0f574e8724cc81: Status 404 returned error can't find the container with id 47145709fb37fa132e8d38fa4cf45749b4bb387bd9e0d221dd0f574e8724cc81
Sep 18 16:23:52.492117 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:52.492032    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c" event={"ID":"4e284191-4656-450d-8bb2-b768ba5a107b","Type":"ContainerStarted","Data":"f7860b28cca5f3796af13235962896e99b01fdc96958831f57b15c814686e55f"}
Sep 18 16:23:52.492117 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:52.492075    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c" event={"ID":"4e284191-4656-450d-8bb2-b768ba5a107b","Type":"ContainerStarted","Data":"47145709fb37fa132e8d38fa4cf45749b4bb387bd9e0d221dd0f574e8724cc81"}
Sep 18 16:23:52.521620 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:52.521568    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c" podStartSLOduration=1.521553134 podStartE2EDuration="1.521553134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:23:52.517628038 +0000 UTC m=+1127.465126814" watchObservedRunningTime="2026-09-18 16:23:52.521553134 +0000 UTC m=+1127.469051908"
Sep 18 16:23:56.674385 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.674352    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"]
Sep 18 16:23:56.676992 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.676975    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.679838 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.679818    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-eab152b6-openshift-default-dockercfg-bl4kc\""
Sep 18 16:23:56.699881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.699848    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"]
Sep 18 16:23:56.810951 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.810905    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-credential-socket\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.810951 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.810954    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-workload-socket\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.811166 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.811033    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52438627-318f-42f1-b2c1-1089381e06c4-istio-token\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.811166 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.811069    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52438627-318f-42f1-b2c1-1089381e06c4-istiod-ca-cert\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.811166 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.811109    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vz9n\" (UniqueName: \"kubernetes.io/projected/52438627-318f-42f1-b2c1-1089381e06c4-kube-api-access-7vz9n\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.811166 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.811157    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52438627-318f-42f1-b2c1-1089381e06c4-istio-podinfo\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.811298 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.811187    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-istio-data\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.811298 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.811215    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-workload-certs\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.811298 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.811260    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-istio-envoy\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.912729 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.912688    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-istio-envoy\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.912729 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.912732    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-credential-socket\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.912997 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.912758    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-workload-socket\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.912997 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.912813    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52438627-318f-42f1-b2c1-1089381e06c4-istio-token\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.912997 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.912844    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52438627-318f-42f1-b2c1-1089381e06c4-istiod-ca-cert\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.912997 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.912886    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vz9n\" (UniqueName: \"kubernetes.io/projected/52438627-318f-42f1-b2c1-1089381e06c4-kube-api-access-7vz9n\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.912997 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.912937    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52438627-318f-42f1-b2c1-1089381e06c4-istio-podinfo\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.912997 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.912973    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-istio-data\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.913360 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.913017    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-workload-certs\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.913360 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.913233    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-credential-socket\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.913472 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.913376    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-workload-certs\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.913472 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.913458    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-istio-data\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.913631 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.913573    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-workload-socket\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.913693 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.913652    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52438627-318f-42f1-b2c1-1089381e06c4-istiod-ca-cert\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.915338 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.915317    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-istio-envoy\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.915575 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.915556    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52438627-318f-42f1-b2c1-1089381e06c4-istio-podinfo\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.925380 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.925322    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vz9n\" (UniqueName: \"kubernetes.io/projected/52438627-318f-42f1-b2c1-1089381e06c4-kube-api-access-7vz9n\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.925480 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.925404    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52438627-318f-42f1-b2c1-1089381e06c4-istio-token\") pod \"e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") " pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:56.988291 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:56.988258    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:57.315843 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:23:57.315804    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52438627_318f_42f1_b2c1_1089381e06c4.slice/crio-11c567a1170cd030d82f7b0104e012e7de9941b00ebc730e7f179c08be0101ff WatchSource:0}: Error finding container 11c567a1170cd030d82f7b0104e012e7de9941b00ebc730e7f179c08be0101ff: Status 404 returned error can't find the container with id 11c567a1170cd030d82f7b0104e012e7de9941b00ebc730e7f179c08be0101ff
Sep 18 16:23:57.317181 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:57.317158    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"]
Sep 18 16:23:57.318424 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:57.318385    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:23:57.318503 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:57.318468    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:23:57.318563 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:57.318507    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:23:57.523914 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:57.523870    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph" event={"ID":"52438627-318f-42f1-b2c1-1089381e06c4","Type":"ContainerStarted","Data":"e7f57599f5efac0d21c87510656b376f4e81839edf2f002e8e9c6f7a50141935"}
Sep 18 16:23:57.524122 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:57.523950    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph" event={"ID":"52438627-318f-42f1-b2c1-1089381e06c4","Type":"ContainerStarted","Data":"11c567a1170cd030d82f7b0104e012e7de9941b00ebc730e7f179c08be0101ff"}
Sep 18 16:23:57.989228 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:57.989192    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:58.993459 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:58.993425    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:59.046747 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:59.046697    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph" podStartSLOduration=3.046682225 podStartE2EDuration="3.046682225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:23:57.582953774 +0000 UTC m=+1132.530452550" watchObservedRunningTime="2026-09-18 16:23:59.046682225 +0000 UTC m=+1133.994180998"
Sep 18 16:23:59.534977 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:59.534947    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:23:59.535881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:23:59.535860    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:24:01.889211 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:01.889177    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:24:01.889619 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:01.889223    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:24:01.901932 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:01.901906    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:24:02.561366 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:02.561337    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:24:30.576720 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:30.576679    2554 patch_prober.go:28] interesting pod/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 18 16:24:30.577142 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:30.576799    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph" podUID="52438627-318f-42f1-b2c1-1089381e06c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:24:31.710560 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:31.710508    2554 generic.go:356] "Generic (PLEG): container finished" podID="52438627-318f-42f1-b2c1-1089381e06c4" containerID="e7f57599f5efac0d21c87510656b376f4e81839edf2f002e8e9c6f7a50141935" exitCode=0
Sep 18 16:24:31.710973 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:31.710574    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph" event={"ID":"52438627-318f-42f1-b2c1-1089381e06c4","Type":"ContainerDied","Data":"e7f57599f5efac0d21c87510656b376f4e81839edf2f002e8e9c6f7a50141935"}
Sep 18 16:24:31.711033 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:31.711018    2554 scope.go:117] "RemoveContainer" containerID="e7f57599f5efac0d21c87510656b376f4e81839edf2f002e8e9c6f7a50141935"
Sep 18 16:24:31.712205 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:31.712162    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:24:31.712310 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:31.712251    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:24:31.712310 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:31.712300    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:24:32.717989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:32.717950    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph" event={"ID":"52438627-318f-42f1-b2c1-1089381e06c4","Type":"ContainerStarted","Data":"876c75f6ceceaf56f5882d80a9f9dd3284e92c0dc8e65bce207675526dcad892"}
Sep 18 16:24:32.989010 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:32.988920    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:24:32.989172 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:32.989150    2554 patch_prober.go:28] interesting pod/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 18 16:24:32.989224 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:32.989203    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph" podUID="52438627-318f-42f1-b2c1-1089381e06c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 18 16:24:33.391050 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:33.390976    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"]
Sep 18 16:24:33.989396 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:33.989361    2554 patch_prober.go:28] interesting pod/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 18 16:24:33.989829 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:33.989420    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph" podUID="52438627-318f-42f1-b2c1-1089381e06c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 18 16:24:34.989417 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:34.989384    2554 patch_prober.go:28] interesting pod/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 18 16:24:34.989861 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:34.989451    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph" podUID="52438627-318f-42f1-b2c1-1089381e06c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 18 16:24:35.988781 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:35.988742    2554 patch_prober.go:28] interesting pod/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 18 16:24:35.988965 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:35.988799    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph" podUID="52438627-318f-42f1-b2c1-1089381e06c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 18 16:24:36.989002 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:36.988962    2554 patch_prober.go:28] interesting pod/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 18 16:24:36.989426 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:36.989027    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph" podUID="52438627-318f-42f1-b2c1-1089381e06c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 18 16:24:37.989058 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:37.989019    2554 patch_prober.go:28] interesting pod/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 18 16:24:37.989471 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:37.989074    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph" podUID="52438627-318f-42f1-b2c1-1089381e06c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 18 16:24:38.992613 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:38.992583    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:24:38.993080 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:38.992845    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:24:38.993554 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:38.993516    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:24:41.431670 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.427934    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"]
Sep 18 16:24:41.431670 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.428243    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8" podUID="71e50642-38b5-480d-bd85-f6846cade66f" containerName="maas-api" containerID="cri-o://148992677b7088bbf10ced978a49489c1c966da958441ce1a9cbe9adabe16bc5" gracePeriod=30
Sep 18 16:24:41.493427 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.493387    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"]
Sep 18 16:24:41.493680 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.493631    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx" podUID="33d8a731-67fd-416f-a0b4-449614660fbe" containerName="payload-processing" containerID="cri-o://871232758cc6f442edf7f6950b6d91ff124a77c64a7037ace4f79e88999c37d8" gracePeriod=30
Sep 18 16:24:41.506942 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.506512    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"]
Sep 18 16:24:41.506942 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.506767    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm" podUID="0d9615e5-ecaf-48dd-a8fb-ce1fd420b101" containerName="payload-pre-processing" containerID="cri-o://923a7aff4bc56e67ffb78dd9d803e2b0e51cc9f5c74875d54caa8b77df4dd118" gracePeriod=30
Sep 18 16:24:41.777342 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.777310    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:24:41.788186 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.788162    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"
Sep 18 16:24:41.788316 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.788225    2554 generic.go:356] "Generic (PLEG): container finished" podID="33d8a731-67fd-416f-a0b4-449614660fbe" containerID="871232758cc6f442edf7f6950b6d91ff124a77c64a7037ace4f79e88999c37d8" exitCode=0
Sep 18 16:24:41.788316 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.788290    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx" event={"ID":"33d8a731-67fd-416f-a0b4-449614660fbe","Type":"ContainerDied","Data":"871232758cc6f442edf7f6950b6d91ff124a77c64a7037ace4f79e88999c37d8"}
Sep 18 16:24:41.788406 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.788342    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx" event={"ID":"33d8a731-67fd-416f-a0b4-449614660fbe","Type":"ContainerDied","Data":"76034f7e01b4e86d1e11f2e88f6cc62c08eb321af6f4c706841596e5d1d482b1"}
Sep 18 16:24:41.788406 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.788372    2554 scope.go:117] "RemoveContainer" containerID="871232758cc6f442edf7f6950b6d91ff124a77c64a7037ace4f79e88999c37d8"
Sep 18 16:24:41.789966 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.789945    2554 generic.go:356] "Generic (PLEG): container finished" podID="0d9615e5-ecaf-48dd-a8fb-ce1fd420b101" containerID="923a7aff4bc56e67ffb78dd9d803e2b0e51cc9f5c74875d54caa8b77df4dd118" exitCode=0
Sep 18 16:24:41.790088 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.790030    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm" event={"ID":"0d9615e5-ecaf-48dd-a8fb-ce1fd420b101","Type":"ContainerDied","Data":"923a7aff4bc56e67ffb78dd9d803e2b0e51cc9f5c74875d54caa8b77df4dd118"}
Sep 18 16:24:41.791265 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.791243    2554 generic.go:356] "Generic (PLEG): container finished" podID="71e50642-38b5-480d-bd85-f6846cade66f" containerID="148992677b7088bbf10ced978a49489c1c966da958441ce1a9cbe9adabe16bc5" exitCode=0
Sep 18 16:24:41.791373 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.791299    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"
Sep 18 16:24:41.791444 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.791297    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8" event={"ID":"71e50642-38b5-480d-bd85-f6846cade66f","Type":"ContainerDied","Data":"148992677b7088bbf10ced978a49489c1c966da958441ce1a9cbe9adabe16bc5"}
Sep 18 16:24:41.791444 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.791409    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8" event={"ID":"71e50642-38b5-480d-bd85-f6846cade66f","Type":"ContainerDied","Data":"5f476972d99cba24d71b7dec8a06200e66e77a445334e0e1c8296e00761d8378"}
Sep 18 16:24:41.797375 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.797344    2554 scope.go:117] "RemoveContainer" containerID="871232758cc6f442edf7f6950b6d91ff124a77c64a7037ace4f79e88999c37d8"
Sep 18 16:24:41.797668 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:24:41.797648    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"871232758cc6f442edf7f6950b6d91ff124a77c64a7037ace4f79e88999c37d8\": container with ID starting with 871232758cc6f442edf7f6950b6d91ff124a77c64a7037ace4f79e88999c37d8 not found: ID does not exist" containerID="871232758cc6f442edf7f6950b6d91ff124a77c64a7037ace4f79e88999c37d8"
Sep 18 16:24:41.797728 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.797678    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871232758cc6f442edf7f6950b6d91ff124a77c64a7037ace4f79e88999c37d8"} err="failed to get container status \"871232758cc6f442edf7f6950b6d91ff124a77c64a7037ace4f79e88999c37d8\": rpc error: code = NotFound desc = could not find container \"871232758cc6f442edf7f6950b6d91ff124a77c64a7037ace4f79e88999c37d8\": container with ID starting with 871232758cc6f442edf7f6950b6d91ff124a77c64a7037ace4f79e88999c37d8 not found: ID does not exist"
Sep 18 16:24:41.797728 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.797695    2554 scope.go:117] "RemoveContainer" containerID="148992677b7088bbf10ced978a49489c1c966da958441ce1a9cbe9adabe16bc5"
Sep 18 16:24:41.807209 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.807189    2554 scope.go:117] "RemoveContainer" containerID="148992677b7088bbf10ced978a49489c1c966da958441ce1a9cbe9adabe16bc5"
Sep 18 16:24:41.807569 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:24:41.807539    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"148992677b7088bbf10ced978a49489c1c966da958441ce1a9cbe9adabe16bc5\": container with ID starting with 148992677b7088bbf10ced978a49489c1c966da958441ce1a9cbe9adabe16bc5 not found: ID does not exist" containerID="148992677b7088bbf10ced978a49489c1c966da958441ce1a9cbe9adabe16bc5"
Sep 18 16:24:41.807671 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.807593    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"148992677b7088bbf10ced978a49489c1c966da958441ce1a9cbe9adabe16bc5"} err="failed to get container status \"148992677b7088bbf10ced978a49489c1c966da958441ce1a9cbe9adabe16bc5\": rpc error: code = NotFound desc = could not find container \"148992677b7088bbf10ced978a49489c1c966da958441ce1a9cbe9adabe16bc5\": container with ID starting with 148992677b7088bbf10ced978a49489c1c966da958441ce1a9cbe9adabe16bc5 not found: ID does not exist"
Sep 18 16:24:41.821485 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.821467    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"
Sep 18 16:24:41.884667 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.884639    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e50642-38b5-480d-bd85-f6846cade66f-openshift-service-ca\") pod \"71e50642-38b5-480d-bd85-f6846cade66f\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") "
Sep 18 16:24:41.884834 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.884734    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cvjn\" (UniqueName: \"kubernetes.io/projected/71e50642-38b5-480d-bd85-f6846cade66f-kube-api-access-8cvjn\") pod \"71e50642-38b5-480d-bd85-f6846cade66f\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") "
Sep 18 16:24:41.884834 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.884760    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e50642-38b5-480d-bd85-f6846cade66f-metrics-certs\") pod \"71e50642-38b5-480d-bd85-f6846cade66f\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") "
Sep 18 16:24:41.884951 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.884836    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e50642-38b5-480d-bd85-f6846cade66f-maas-api-tls\") pod \"71e50642-38b5-480d-bd85-f6846cade66f\" (UID: \"71e50642-38b5-480d-bd85-f6846cade66f\") "
Sep 18 16:24:41.885098 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.885070    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e50642-38b5-480d-bd85-f6846cade66f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71e50642-38b5-480d-bd85-f6846cade66f" (UID: "71e50642-38b5-480d-bd85-f6846cade66f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:24:41.885215 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.885193    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e50642-38b5-480d-bd85-f6846cade66f-openshift-service-ca\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:41.887303 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.887281    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e50642-38b5-480d-bd85-f6846cade66f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71e50642-38b5-480d-bd85-f6846cade66f" (UID: "71e50642-38b5-480d-bd85-f6846cade66f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:24:41.887632 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.887608    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e50642-38b5-480d-bd85-f6846cade66f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "71e50642-38b5-480d-bd85-f6846cade66f" (UID: "71e50642-38b5-480d-bd85-f6846cade66f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:24:41.887709 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.887625    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e50642-38b5-480d-bd85-f6846cade66f-kube-api-access-8cvjn" (OuterVolumeSpecName: "kube-api-access-8cvjn") pod "71e50642-38b5-480d-bd85-f6846cade66f" (UID: "71e50642-38b5-480d-bd85-f6846cade66f"). InnerVolumeSpecName "kube-api-access-8cvjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:24:41.985597 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.985490    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxkrh\" (UniqueName: \"kubernetes.io/projected/33d8a731-67fd-416f-a0b4-449614660fbe-kube-api-access-sxkrh\") pod \"33d8a731-67fd-416f-a0b4-449614660fbe\" (UID: \"33d8a731-67fd-416f-a0b4-449614660fbe\") "
Sep 18 16:24:41.985597 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.985557    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d8a731-67fd-416f-a0b4-449614660fbe-plugins-config-volume\") pod \"33d8a731-67fd-416f-a0b4-449614660fbe\" (UID: \"33d8a731-67fd-416f-a0b4-449614660fbe\") "
Sep 18 16:24:41.985831 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.985628    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d9615e5-ecaf-48dd-a8fb-ce1fd420b101-plugins-config-volume\") pod \"0d9615e5-ecaf-48dd-a8fb-ce1fd420b101\" (UID: \"0d9615e5-ecaf-48dd-a8fb-ce1fd420b101\") "
Sep 18 16:24:41.985831 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.985701    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpsgh\" (UniqueName: \"kubernetes.io/projected/0d9615e5-ecaf-48dd-a8fb-ce1fd420b101-kube-api-access-jpsgh\") pod \"0d9615e5-ecaf-48dd-a8fb-ce1fd420b101\" (UID: \"0d9615e5-ecaf-48dd-a8fb-ce1fd420b101\") "
Sep 18 16:24:41.985941 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.985887    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cvjn\" (UniqueName: \"kubernetes.io/projected/71e50642-38b5-480d-bd85-f6846cade66f-kube-api-access-8cvjn\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:41.985941 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.985903    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e50642-38b5-480d-bd85-f6846cade66f-metrics-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:41.985941 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.985916    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e50642-38b5-480d-bd85-f6846cade66f-maas-api-tls\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:41.986049 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.985968    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d8a731-67fd-416f-a0b4-449614660fbe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33d8a731-67fd-416f-a0b4-449614660fbe" (UID: "33d8a731-67fd-416f-a0b4-449614660fbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:24:41.986049 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.986033    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d9615e5-ecaf-48dd-a8fb-ce1fd420b101-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d9615e5-ecaf-48dd-a8fb-ce1fd420b101" (UID: "0d9615e5-ecaf-48dd-a8fb-ce1fd420b101"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:24:41.987965 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.987942    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d9615e5-ecaf-48dd-a8fb-ce1fd420b101-kube-api-access-jpsgh" (OuterVolumeSpecName: "kube-api-access-jpsgh") pod "0d9615e5-ecaf-48dd-a8fb-ce1fd420b101" (UID: "0d9615e5-ecaf-48dd-a8fb-ce1fd420b101"). InnerVolumeSpecName "kube-api-access-jpsgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:24:41.988071 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:41.987969    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d8a731-67fd-416f-a0b4-449614660fbe-kube-api-access-sxkrh" (OuterVolumeSpecName: "kube-api-access-sxkrh") pod "33d8a731-67fd-416f-a0b4-449614660fbe" (UID: "33d8a731-67fd-416f-a0b4-449614660fbe"). InnerVolumeSpecName "kube-api-access-sxkrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:24:42.086946 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:42.086910    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d9615e5-ecaf-48dd-a8fb-ce1fd420b101-plugins-config-volume\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:42.086946 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:42.086941    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpsgh\" (UniqueName: \"kubernetes.io/projected/0d9615e5-ecaf-48dd-a8fb-ce1fd420b101-kube-api-access-jpsgh\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:42.086946 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:42.086953    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxkrh\" (UniqueName: \"kubernetes.io/projected/33d8a731-67fd-416f-a0b4-449614660fbe-kube-api-access-sxkrh\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:42.087179 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:42.086961    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d8a731-67fd-416f-a0b4-449614660fbe-plugins-config-volume\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:42.117770 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:42.117739    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"]
Sep 18 16:24:42.134601 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:42.134573    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b73e26-58bf96bc-mvzj8"]
Sep 18 16:24:42.798432 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:42.798343    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"
Sep 18 16:24:42.799599 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:42.799569    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm" event={"ID":"0d9615e5-ecaf-48dd-a8fb-ce1fd420b101","Type":"ContainerDied","Data":"d688191a50f0499e53d63d20b7922a1c70da98d93ca86102a26faf61f4545a14"}
Sep 18 16:24:42.799752 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:42.799602    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"
Sep 18 16:24:42.799752 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:42.799639    2554 scope.go:117] "RemoveContainer" containerID="923a7aff4bc56e67ffb78dd9d803e2b0e51cc9f5c74875d54caa8b77df4dd118"
Sep 18 16:24:42.825704 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:42.825676    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"]
Sep 18 16:24:42.833084 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:42.833061    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b73e26-79fd96485d-2q6qm"]
Sep 18 16:24:42.845479 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:42.845457    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"]
Sep 18 16:24:42.849814 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:42.849793    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b73e26-7987d9b777-z7wkx"]
Sep 18 16:24:43.278392 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:43.278358    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d9615e5-ecaf-48dd-a8fb-ce1fd420b101" path="/var/lib/kubelet/pods/0d9615e5-ecaf-48dd-a8fb-ce1fd420b101/volumes"
Sep 18 16:24:43.278763 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:43.278749    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33d8a731-67fd-416f-a0b4-449614660fbe" path="/var/lib/kubelet/pods/33d8a731-67fd-416f-a0b4-449614660fbe/volumes"
Sep 18 16:24:43.279096 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:43.279084    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71e50642-38b5-480d-bd85-f6846cade66f" path="/var/lib/kubelet/pods/71e50642-38b5-480d-bd85-f6846cade66f/volumes"
Sep 18 16:24:48.410649 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.410577    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c" podUID="4e284191-4656-450d-8bb2-b768ba5a107b" containerName="main" containerID="cri-o://f7860b28cca5f3796af13235962896e99b01fdc96958831f57b15c814686e55f" gracePeriod=45
Sep 18 16:24:48.663752 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.663681    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:24:48.740633 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.740599    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e284191-4656-450d-8bb2-b768ba5a107b-tls-certs\") pod \"4e284191-4656-450d-8bb2-b768ba5a107b\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") "
Sep 18 16:24:48.740633 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.740638    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-model-cache\") pod \"4e284191-4656-450d-8bb2-b768ba5a107b\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") "
Sep 18 16:24:48.740875 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.740695    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-home\") pod \"4e284191-4656-450d-8bb2-b768ba5a107b\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") "
Sep 18 16:24:48.740875 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.740730    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-dshm\") pod \"4e284191-4656-450d-8bb2-b768ba5a107b\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") "
Sep 18 16:24:48.740875 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.740770    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-tmp-dir\") pod \"4e284191-4656-450d-8bb2-b768ba5a107b\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") "
Sep 18 16:24:48.740875 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.740798    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bf6bg\" (UniqueName: \"kubernetes.io/projected/4e284191-4656-450d-8bb2-b768ba5a107b-kube-api-access-bf6bg\") pod \"4e284191-4656-450d-8bb2-b768ba5a107b\" (UID: \"4e284191-4656-450d-8bb2-b768ba5a107b\") "
Sep 18 16:24:48.741088 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.740934    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e284191-4656-450d-8bb2-b768ba5a107b" (UID: "4e284191-4656-450d-8bb2-b768ba5a107b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:24:48.741088 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.740946    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-home" (OuterVolumeSpecName: "home") pod "4e284191-4656-450d-8bb2-b768ba5a107b" (UID: "4e284191-4656-450d-8bb2-b768ba5a107b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:24:48.741088 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.741052    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e284191-4656-450d-8bb2-b768ba5a107b" (UID: "4e284191-4656-450d-8bb2-b768ba5a107b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:24:48.741224 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.741117    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-model-cache\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:48.741224 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.741136    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-home\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:48.743081 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.743048    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e284191-4656-450d-8bb2-b768ba5a107b-kube-api-access-bf6bg" (OuterVolumeSpecName: "kube-api-access-bf6bg") pod "4e284191-4656-450d-8bb2-b768ba5a107b" (UID: "4e284191-4656-450d-8bb2-b768ba5a107b"). InnerVolumeSpecName "kube-api-access-bf6bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:24:48.743081 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.743074    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e284191-4656-450d-8bb2-b768ba5a107b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e284191-4656-450d-8bb2-b768ba5a107b" (UID: "4e284191-4656-450d-8bb2-b768ba5a107b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:24:48.743233 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.743084    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-dshm" (OuterVolumeSpecName: "dshm") pod "4e284191-4656-450d-8bb2-b768ba5a107b" (UID: "4e284191-4656-450d-8bb2-b768ba5a107b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:24:48.833981 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.833942    2554 generic.go:356] "Generic (PLEG): container finished" podID="4e284191-4656-450d-8bb2-b768ba5a107b" containerID="f7860b28cca5f3796af13235962896e99b01fdc96958831f57b15c814686e55f" exitCode=0
Sep 18 16:24:48.834175 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.834002    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"
Sep 18 16:24:48.834175 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.834028    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c" event={"ID":"4e284191-4656-450d-8bb2-b768ba5a107b","Type":"ContainerDied","Data":"f7860b28cca5f3796af13235962896e99b01fdc96958831f57b15c814686e55f"}
Sep 18 16:24:48.834175 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.834073    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c" event={"ID":"4e284191-4656-450d-8bb2-b768ba5a107b","Type":"ContainerDied","Data":"47145709fb37fa132e8d38fa4cf45749b4bb387bd9e0d221dd0f574e8724cc81"}
Sep 18 16:24:48.834175 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.834097    2554 scope.go:117] "RemoveContainer" containerID="f7860b28cca5f3796af13235962896e99b01fdc96958831f57b15c814686e55f"
Sep 18 16:24:48.842312 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.842067    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e284191-4656-450d-8bb2-b768ba5a107b-tls-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:48.842312 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.842092    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-dshm\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:48.842312 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.842107    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e284191-4656-450d-8bb2-b768ba5a107b-tmp-dir\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:48.842312 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.842120    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf6bg\" (UniqueName: \"kubernetes.io/projected/4e284191-4656-450d-8bb2-b768ba5a107b-kube-api-access-bf6bg\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:48.844249 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.844228    2554 scope.go:117] "RemoveContainer" containerID="f7860b28cca5f3796af13235962896e99b01fdc96958831f57b15c814686e55f"
Sep 18 16:24:48.844600 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:24:48.844578    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7860b28cca5f3796af13235962896e99b01fdc96958831f57b15c814686e55f\": container with ID starting with f7860b28cca5f3796af13235962896e99b01fdc96958831f57b15c814686e55f not found: ID does not exist" containerID="f7860b28cca5f3796af13235962896e99b01fdc96958831f57b15c814686e55f"
Sep 18 16:24:48.844691 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.844612    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7860b28cca5f3796af13235962896e99b01fdc96958831f57b15c814686e55f"} err="failed to get container status \"f7860b28cca5f3796af13235962896e99b01fdc96958831f57b15c814686e55f\": rpc error: code = NotFound desc = could not find container \"f7860b28cca5f3796af13235962896e99b01fdc96958831f57b15c814686e55f\": container with ID starting with f7860b28cca5f3796af13235962896e99b01fdc96958831f57b15c814686e55f not found: ID does not exist"
Sep 18 16:24:48.860765 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.860735    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"]
Sep 18 16:24:48.867146 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:48.867117    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-b73e26/explicit-model-b73e26-kserve-c88b59747-5nm4c"]
Sep 18 16:24:49.278418 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:49.278372    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e284191-4656-450d-8bb2-b768ba5a107b" path="/var/lib/kubelet/pods/4e284191-4656-450d-8bb2-b768ba5a107b/volumes"
Sep 18 16:24:54.085440 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.085401    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"]
Sep 18 16:24:54.085947 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.085697    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q" podUID="a7d69023-7ac5-432b-982c-f4ef453e0103" containerName="istio-proxy" containerID="cri-o://41a10eee61abaa2e5aed1d412ee46d3fb42e7c0bec6163d9e9d0e44fc5036135" gracePeriod=30
Sep 18 16:24:54.830816 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.830783    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"]
Sep 18 16:24:54.831269 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.831251    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e284191-4656-450d-8bb2-b768ba5a107b" containerName="main"
Sep 18 16:24:54.831359 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.831272    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e284191-4656-450d-8bb2-b768ba5a107b" containerName="main"
Sep 18 16:24:54.831359 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.831292    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33d8a731-67fd-416f-a0b4-449614660fbe" containerName="payload-processing"
Sep 18 16:24:54.831359 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.831301    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="33d8a731-67fd-416f-a0b4-449614660fbe" containerName="payload-processing"
Sep 18 16:24:54.831359 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.831310    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d9615e5-ecaf-48dd-a8fb-ce1fd420b101" containerName="payload-pre-processing"
Sep 18 16:24:54.831359 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.831320    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d9615e5-ecaf-48dd-a8fb-ce1fd420b101" containerName="payload-pre-processing"
Sep 18 16:24:54.831359 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.831347    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e50642-38b5-480d-bd85-f6846cade66f" containerName="maas-api"
Sep 18 16:24:54.831359 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.831355    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e50642-38b5-480d-bd85-f6846cade66f" containerName="maas-api"
Sep 18 16:24:54.831759 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.831464    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e50642-38b5-480d-bd85-f6846cade66f" containerName="maas-api"
Sep 18 16:24:54.831759 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.831475    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="33d8a731-67fd-416f-a0b4-449614660fbe" containerName="payload-processing"
Sep 18 16:24:54.831759 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.831491    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e284191-4656-450d-8bb2-b768ba5a107b" containerName="main"
Sep 18 16:24:54.831759 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.831502    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d9615e5-ecaf-48dd-a8fb-ce1fd420b101" containerName="payload-pre-processing"
Sep 18 16:24:54.833540 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.833506    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:54.841180 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.841164    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-72547f-openshift-default-dockercfg-96r8n\""
Sep 18 16:24:54.853451 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.853430    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"]
Sep 18 16:24:54.898741 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.898718    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-credential-socket\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:54.898851 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.898753    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfgz\" (UniqueName: \"kubernetes.io/projected/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-kube-api-access-9pfgz\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:54.898851 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.898772    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-workload-socket\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:54.898932 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.898852    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-envoy\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:54.898932 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.898889    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-podinfo\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:54.898932 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.898911    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-data\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:54.899093 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.898995    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-workload-certs\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:54.899093 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.899041    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istiod-ca-cert\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:54.899186 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:54.899113    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-token\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.000484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.000433    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-envoy\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.000484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.000478    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-podinfo\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.000814 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.000500    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-data\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.000814 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.000560    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-workload-certs\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.000814 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.000588    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istiod-ca-cert\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.001000 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.000984    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-data\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.001055 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.001001    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-workload-certs\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.001055 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.001043    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-token\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.001202 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.001113    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-credential-socket\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.001202 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.001150    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfgz\" (UniqueName: \"kubernetes.io/projected/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-kube-api-access-9pfgz\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.001202 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.001175    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-workload-socket\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.001547 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.001417    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-credential-socket\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.001547 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.001477    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-workload-socket\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.001547 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.001492    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istiod-ca-cert\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.003133 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.003111    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-envoy\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.003267 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.003251    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-podinfo\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.011467 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.011441    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-token\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.013839 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.013814    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfgz\" (UniqueName: \"kubernetes.io/projected/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-kube-api-access-9pfgz\") pod \"e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") " pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.143887 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.143797    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:55.425265 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.425234    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"]
Sep 18 16:24:55.426781 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:24:55.426753    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fd3601_b37f_4446_a364_b1c1fd98cf4c.slice/crio-3a298a269bae5e4331eaea3a7772f2f03e8339a71a68f60e0246d96402fd13aa WatchSource:0}: Error finding container 3a298a269bae5e4331eaea3a7772f2f03e8339a71a68f60e0246d96402fd13aa: Status 404 returned error can't find the container with id 3a298a269bae5e4331eaea3a7772f2f03e8339a71a68f60e0246d96402fd13aa
Sep 18 16:24:55.428991 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.428958    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:24:55.429066 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.429022    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:24:55.429066 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.429051    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:24:55.873022 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.872987    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz" event={"ID":"f2fd3601-b37f-4446-a364-b1c1fd98cf4c","Type":"ContainerStarted","Data":"64e91f85e2f6c078f812efd31294292e1d89b58c7585081ca82ffff636296129"}
Sep 18 16:24:55.873168 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.873037    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz" event={"ID":"f2fd3601-b37f-4446-a364-b1c1fd98cf4c","Type":"ContainerStarted","Data":"3a298a269bae5e4331eaea3a7772f2f03e8339a71a68f60e0246d96402fd13aa"}
Sep 18 16:24:55.903243 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:55.903197    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz" podStartSLOduration=1.903182404 podStartE2EDuration="1.903182404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:24:55.900448799 +0000 UTC m=+1190.847947574" watchObservedRunningTime="2026-09-18 16:24:55.903182404 +0000 UTC m=+1190.850681207"
Sep 18 16:24:56.144378 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:56.144296    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:56.149131 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:56.149104    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:56.880085 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:56.879960    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:56.885804 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:56.885771    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:24:59.334114 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.334090    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:24:59.446773 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.446743    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-envoy\") pod \"a7d69023-7ac5-432b-982c-f4ef453e0103\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") "
Sep 18 16:24:59.446957 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.446788    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h9cb\" (UniqueName: \"kubernetes.io/projected/a7d69023-7ac5-432b-982c-f4ef453e0103-kube-api-access-2h9cb\") pod \"a7d69023-7ac5-432b-982c-f4ef453e0103\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") "
Sep 18 16:24:59.446957 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.446822    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d69023-7ac5-432b-982c-f4ef453e0103-istiod-ca-cert\") pod \"a7d69023-7ac5-432b-982c-f4ef453e0103\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") "
Sep 18 16:24:59.446957 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.446850    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-credential-socket\") pod \"a7d69023-7ac5-432b-982c-f4ef453e0103\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") "
Sep 18 16:24:59.446957 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.446897    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-workload-certs\") pod \"a7d69023-7ac5-432b-982c-f4ef453e0103\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") "
Sep 18 16:24:59.446957 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.446942    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-podinfo\") pod \"a7d69023-7ac5-432b-982c-f4ef453e0103\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") "
Sep 18 16:24:59.447208 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.446970    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-data\") pod \"a7d69023-7ac5-432b-982c-f4ef453e0103\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") "
Sep 18 16:24:59.447208 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.447012    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-workload-socket\") pod \"a7d69023-7ac5-432b-982c-f4ef453e0103\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") "
Sep 18 16:24:59.447208 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.447047    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-token\") pod \"a7d69023-7ac5-432b-982c-f4ef453e0103\" (UID: \"a7d69023-7ac5-432b-982c-f4ef453e0103\") "
Sep 18 16:24:59.447208 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.447188    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7d69023-7ac5-432b-982c-f4ef453e0103" (UID: "a7d69023-7ac5-432b-982c-f4ef453e0103"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:24:59.447406 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.447209    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7d69023-7ac5-432b-982c-f4ef453e0103" (UID: "a7d69023-7ac5-432b-982c-f4ef453e0103"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:24:59.447406 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.447281    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d69023-7ac5-432b-982c-f4ef453e0103-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7d69023-7ac5-432b-982c-f4ef453e0103" (UID: "a7d69023-7ac5-432b-982c-f4ef453e0103"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:24:59.447584 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.447557    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7d69023-7ac5-432b-982c-f4ef453e0103" (UID: "a7d69023-7ac5-432b-982c-f4ef453e0103"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:24:59.447804 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.447693    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7d69023-7ac5-432b-982c-f4ef453e0103" (UID: "a7d69023-7ac5-432b-982c-f4ef453e0103"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:24:59.447941 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.447753    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d69023-7ac5-432b-982c-f4ef453e0103-istiod-ca-cert\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:59.448101 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.447962    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-credential-socket\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:59.448101 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.447977    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-workload-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:59.449404 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.449380    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7d69023-7ac5-432b-982c-f4ef453e0103" (UID: "a7d69023-7ac5-432b-982c-f4ef453e0103"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:24:59.449503 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.449462    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d69023-7ac5-432b-982c-f4ef453e0103-kube-api-access-2h9cb" (OuterVolumeSpecName: "kube-api-access-2h9cb") pod "a7d69023-7ac5-432b-982c-f4ef453e0103" (UID: "a7d69023-7ac5-432b-982c-f4ef453e0103"). InnerVolumeSpecName "kube-api-access-2h9cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:24:59.449704 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.449678    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7d69023-7ac5-432b-982c-f4ef453e0103" (UID: "a7d69023-7ac5-432b-982c-f4ef453e0103"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:24:59.449859 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.449840    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7d69023-7ac5-432b-982c-f4ef453e0103" (UID: "a7d69023-7ac5-432b-982c-f4ef453e0103"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:24:59.549231 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.549196    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-workload-socket\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:59.549231 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.549225    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-token\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:59.549231 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.549233    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-envoy\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:59.549467 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.549242    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h9cb\" (UniqueName: \"kubernetes.io/projected/a7d69023-7ac5-432b-982c-f4ef453e0103-kube-api-access-2h9cb\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:59.549467 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.549249    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-podinfo\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:59.549467 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.549257    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d69023-7ac5-432b-982c-f4ef453e0103-istio-data\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:24:59.896237 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.896135    2554 generic.go:356] "Generic (PLEG): container finished" podID="a7d69023-7ac5-432b-982c-f4ef453e0103" containerID="41a10eee61abaa2e5aed1d412ee46d3fb42e7c0bec6163d9e9d0e44fc5036135" exitCode=0
Sep 18 16:24:59.896237 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.896195    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"
Sep 18 16:24:59.896237 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.896219    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q" event={"ID":"a7d69023-7ac5-432b-982c-f4ef453e0103","Type":"ContainerDied","Data":"41a10eee61abaa2e5aed1d412ee46d3fb42e7c0bec6163d9e9d0e44fc5036135"}
Sep 18 16:24:59.896557 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.896269    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q" event={"ID":"a7d69023-7ac5-432b-982c-f4ef453e0103","Type":"ContainerDied","Data":"71ceb4cfbe7d3a4182a76da1fe5c4f79a462eb0ae52f8136103e9b9c226b8476"}
Sep 18 16:24:59.896557 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.896293    2554 scope.go:117] "RemoveContainer" containerID="41a10eee61abaa2e5aed1d412ee46d3fb42e7c0bec6163d9e9d0e44fc5036135"
Sep 18 16:24:59.906279 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.906264    2554 scope.go:117] "RemoveContainer" containerID="11f6f621de22e1e39c8bb94d65b8c274696c27bdd3dab6bad61c1d1a7d9bb3fe"
Sep 18 16:24:59.915094 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.915075    2554 scope.go:117] "RemoveContainer" containerID="41a10eee61abaa2e5aed1d412ee46d3fb42e7c0bec6163d9e9d0e44fc5036135"
Sep 18 16:24:59.915401 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:24:59.915382    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a10eee61abaa2e5aed1d412ee46d3fb42e7c0bec6163d9e9d0e44fc5036135\": container with ID starting with 41a10eee61abaa2e5aed1d412ee46d3fb42e7c0bec6163d9e9d0e44fc5036135 not found: ID does not exist" containerID="41a10eee61abaa2e5aed1d412ee46d3fb42e7c0bec6163d9e9d0e44fc5036135"
Sep 18 16:24:59.915475 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.915415    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a10eee61abaa2e5aed1d412ee46d3fb42e7c0bec6163d9e9d0e44fc5036135"} err="failed to get container status \"41a10eee61abaa2e5aed1d412ee46d3fb42e7c0bec6163d9e9d0e44fc5036135\": rpc error: code = NotFound desc = could not find container \"41a10eee61abaa2e5aed1d412ee46d3fb42e7c0bec6163d9e9d0e44fc5036135\": container with ID starting with 41a10eee61abaa2e5aed1d412ee46d3fb42e7c0bec6163d9e9d0e44fc5036135 not found: ID does not exist"
Sep 18 16:24:59.915475 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.915438    2554 scope.go:117] "RemoveContainer" containerID="11f6f621de22e1e39c8bb94d65b8c274696c27bdd3dab6bad61c1d1a7d9bb3fe"
Sep 18 16:24:59.915753 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:24:59.915730    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f6f621de22e1e39c8bb94d65b8c274696c27bdd3dab6bad61c1d1a7d9bb3fe\": container with ID starting with 11f6f621de22e1e39c8bb94d65b8c274696c27bdd3dab6bad61c1d1a7d9bb3fe not found: ID does not exist" containerID="11f6f621de22e1e39c8bb94d65b8c274696c27bdd3dab6bad61c1d1a7d9bb3fe"
Sep 18 16:24:59.915797 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.915759    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f6f621de22e1e39c8bb94d65b8c274696c27bdd3dab6bad61c1d1a7d9bb3fe"} err="failed to get container status \"11f6f621de22e1e39c8bb94d65b8c274696c27bdd3dab6bad61c1d1a7d9bb3fe\": rpc error: code = NotFound desc = could not find container \"11f6f621de22e1e39c8bb94d65b8c274696c27bdd3dab6bad61c1d1a7d9bb3fe\": container with ID starting with 11f6f621de22e1e39c8bb94d65b8c274696c27bdd3dab6bad61c1d1a7d9bb3fe not found: ID does not exist"
Sep 18 16:24:59.920274 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.920256    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"]
Sep 18 16:24:59.924040 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:24:59.924021    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-b73e26-openshift-default-6dd9f46656-g6f9q"]
Sep 18 16:25:01.277093 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:01.277057    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7d69023-7ac5-432b-982c-f4ef453e0103" path="/var/lib/kubelet/pods/a7d69023-7ac5-432b-982c-f4ef453e0103/volumes"
Sep 18 16:25:03.443984 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:03.443950    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"]
Sep 18 16:25:03.444422 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:03.444200    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph" podUID="52438627-318f-42f1-b2c1-1089381e06c4" containerName="istio-proxy" containerID="cri-o://876c75f6ceceaf56f5882d80a9f9dd3284e92c0dc8e65bce207675526dcad892" gracePeriod=30
Sep 18 16:25:08.699639 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.699611    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:25:08.838091 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.837989    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vz9n\" (UniqueName: \"kubernetes.io/projected/52438627-318f-42f1-b2c1-1089381e06c4-kube-api-access-7vz9n\") pod \"52438627-318f-42f1-b2c1-1089381e06c4\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") "
Sep 18 16:25:08.838091 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.838038    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-istio-envoy\") pod \"52438627-318f-42f1-b2c1-1089381e06c4\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") "
Sep 18 16:25:08.838091 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.838073    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-workload-certs\") pod \"52438627-318f-42f1-b2c1-1089381e06c4\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") "
Sep 18 16:25:08.838379 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.838100    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52438627-318f-42f1-b2c1-1089381e06c4-istio-token\") pod \"52438627-318f-42f1-b2c1-1089381e06c4\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") "
Sep 18 16:25:08.838379 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.838121    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-credential-socket\") pod \"52438627-318f-42f1-b2c1-1089381e06c4\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") "
Sep 18 16:25:08.838379 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.838162    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-workload-socket\") pod \"52438627-318f-42f1-b2c1-1089381e06c4\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") "
Sep 18 16:25:08.838379 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.838188    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52438627-318f-42f1-b2c1-1089381e06c4-istio-podinfo\") pod \"52438627-318f-42f1-b2c1-1089381e06c4\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") "
Sep 18 16:25:08.838379 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.838208    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-istio-data\") pod \"52438627-318f-42f1-b2c1-1089381e06c4\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") "
Sep 18 16:25:08.838379 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.838264    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52438627-318f-42f1-b2c1-1089381e06c4-istiod-ca-cert\") pod \"52438627-318f-42f1-b2c1-1089381e06c4\" (UID: \"52438627-318f-42f1-b2c1-1089381e06c4\") "
Sep 18 16:25:08.838719 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.838445    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "52438627-318f-42f1-b2c1-1089381e06c4" (UID: "52438627-318f-42f1-b2c1-1089381e06c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:25:08.838719 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.838463    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "52438627-318f-42f1-b2c1-1089381e06c4" (UID: "52438627-318f-42f1-b2c1-1089381e06c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:25:08.838719 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.838472    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "52438627-318f-42f1-b2c1-1089381e06c4" (UID: "52438627-318f-42f1-b2c1-1089381e06c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:25:08.838719 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.838705    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "52438627-318f-42f1-b2c1-1089381e06c4" (UID: "52438627-318f-42f1-b2c1-1089381e06c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:25:08.838883 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.838731    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-credential-socket\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:08.838883 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.838753    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-workload-socket\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:08.838883 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.838769    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-workload-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:08.838883 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.838818    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52438627-318f-42f1-b2c1-1089381e06c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "52438627-318f-42f1-b2c1-1089381e06c4" (UID: "52438627-318f-42f1-b2c1-1089381e06c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:25:08.840432 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.840407    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52438627-318f-42f1-b2c1-1089381e06c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "52438627-318f-42f1-b2c1-1089381e06c4" (UID: "52438627-318f-42f1-b2c1-1089381e06c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:25:08.840559 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.840430    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "52438627-318f-42f1-b2c1-1089381e06c4" (UID: "52438627-318f-42f1-b2c1-1089381e06c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:25:08.841042 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.841020    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52438627-318f-42f1-b2c1-1089381e06c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "52438627-318f-42f1-b2c1-1089381e06c4" (UID: "52438627-318f-42f1-b2c1-1089381e06c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:25:08.841125 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.841024    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52438627-318f-42f1-b2c1-1089381e06c4-kube-api-access-7vz9n" (OuterVolumeSpecName: "kube-api-access-7vz9n") pod "52438627-318f-42f1-b2c1-1089381e06c4" (UID: "52438627-318f-42f1-b2c1-1089381e06c4"). InnerVolumeSpecName "kube-api-access-7vz9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:25:08.939312 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.939282    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52438627-318f-42f1-b2c1-1089381e06c4-istio-podinfo\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:08.939312 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.939307    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-istio-data\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:08.939312 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.939315    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52438627-318f-42f1-b2c1-1089381e06c4-istiod-ca-cert\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:08.939570 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.939324    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vz9n\" (UniqueName: \"kubernetes.io/projected/52438627-318f-42f1-b2c1-1089381e06c4-kube-api-access-7vz9n\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:08.939570 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.939334    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52438627-318f-42f1-b2c1-1089381e06c4-istio-envoy\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:08.939570 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.939342    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52438627-318f-42f1-b2c1-1089381e06c4-istio-token\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:08.943849 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.943814    2554 generic.go:356] "Generic (PLEG): container finished" podID="52438627-318f-42f1-b2c1-1089381e06c4" containerID="876c75f6ceceaf56f5882d80a9f9dd3284e92c0dc8e65bce207675526dcad892" exitCode=0
Sep 18 16:25:08.943971 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.943852    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph" event={"ID":"52438627-318f-42f1-b2c1-1089381e06c4","Type":"ContainerDied","Data":"876c75f6ceceaf56f5882d80a9f9dd3284e92c0dc8e65bce207675526dcad892"}
Sep 18 16:25:08.943971 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.943862    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"
Sep 18 16:25:08.943971 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.943909    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph" event={"ID":"52438627-318f-42f1-b2c1-1089381e06c4","Type":"ContainerDied","Data":"11c567a1170cd030d82f7b0104e012e7de9941b00ebc730e7f179c08be0101ff"}
Sep 18 16:25:08.943971 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.943940    2554 scope.go:117] "RemoveContainer" containerID="876c75f6ceceaf56f5882d80a9f9dd3284e92c0dc8e65bce207675526dcad892"
Sep 18 16:25:08.953462 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.953441    2554 scope.go:117] "RemoveContainer" containerID="e7f57599f5efac0d21c87510656b376f4e81839edf2f002e8e9c6f7a50141935"
Sep 18 16:25:08.963811 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.963791    2554 scope.go:117] "RemoveContainer" containerID="876c75f6ceceaf56f5882d80a9f9dd3284e92c0dc8e65bce207675526dcad892"
Sep 18 16:25:08.964215 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:25:08.964195    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876c75f6ceceaf56f5882d80a9f9dd3284e92c0dc8e65bce207675526dcad892\": container with ID starting with 876c75f6ceceaf56f5882d80a9f9dd3284e92c0dc8e65bce207675526dcad892 not found: ID does not exist" containerID="876c75f6ceceaf56f5882d80a9f9dd3284e92c0dc8e65bce207675526dcad892"
Sep 18 16:25:08.964275 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.964225    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876c75f6ceceaf56f5882d80a9f9dd3284e92c0dc8e65bce207675526dcad892"} err="failed to get container status \"876c75f6ceceaf56f5882d80a9f9dd3284e92c0dc8e65bce207675526dcad892\": rpc error: code = NotFound desc = could not find container \"876c75f6ceceaf56f5882d80a9f9dd3284e92c0dc8e65bce207675526dcad892\": container with ID starting with 876c75f6ceceaf56f5882d80a9f9dd3284e92c0dc8e65bce207675526dcad892 not found: ID does not exist"
Sep 18 16:25:08.964275 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.964246    2554 scope.go:117] "RemoveContainer" containerID="e7f57599f5efac0d21c87510656b376f4e81839edf2f002e8e9c6f7a50141935"
Sep 18 16:25:08.964514 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:25:08.964500    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f57599f5efac0d21c87510656b376f4e81839edf2f002e8e9c6f7a50141935\": container with ID starting with e7f57599f5efac0d21c87510656b376f4e81839edf2f002e8e9c6f7a50141935 not found: ID does not exist" containerID="e7f57599f5efac0d21c87510656b376f4e81839edf2f002e8e9c6f7a50141935"
Sep 18 16:25:08.964621 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.964547    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f57599f5efac0d21c87510656b376f4e81839edf2f002e8e9c6f7a50141935"} err="failed to get container status \"e7f57599f5efac0d21c87510656b376f4e81839edf2f002e8e9c6f7a50141935\": rpc error: code = NotFound desc = could not find container \"e7f57599f5efac0d21c87510656b376f4e81839edf2f002e8e9c6f7a50141935\": container with ID starting with e7f57599f5efac0d21c87510656b376f4e81839edf2f002e8e9c6f7a50141935 not found: ID does not exist"
Sep 18 16:25:08.969272 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.969249    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"]
Sep 18 16:25:08.978081 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:08.978046    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-eab152b6-openshift-default-c57fc98bd-d6gph"]
Sep 18 16:25:09.278660 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:09.278626    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52438627-318f-42f1-b2c1-1089381e06c4" path="/var/lib/kubelet/pods/52438627-318f-42f1-b2c1-1089381e06c4/volumes"
Sep 18 16:25:14.961500 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.961467    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"]
Sep 18 16:25:14.961965 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.961875    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7d69023-7ac5-432b-982c-f4ef453e0103" containerName="istio-proxy"
Sep 18 16:25:14.961965 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.961886    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7d69023-7ac5-432b-982c-f4ef453e0103" containerName="istio-proxy"
Sep 18 16:25:14.961965 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.961900    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7d69023-7ac5-432b-982c-f4ef453e0103" containerName="istio-proxy"
Sep 18 16:25:14.961965 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.961905    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7d69023-7ac5-432b-982c-f4ef453e0103" containerName="istio-proxy"
Sep 18 16:25:14.961965 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.961922    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52438627-318f-42f1-b2c1-1089381e06c4" containerName="istio-proxy"
Sep 18 16:25:14.961965 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.961927    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="52438627-318f-42f1-b2c1-1089381e06c4" containerName="istio-proxy"
Sep 18 16:25:14.962193 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.961991    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="52438627-318f-42f1-b2c1-1089381e06c4" containerName="istio-proxy"
Sep 18 16:25:14.962193 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.961998    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7d69023-7ac5-432b-982c-f4ef453e0103" containerName="istio-proxy"
Sep 18 16:25:14.962193 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.962004    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="52438627-318f-42f1-b2c1-1089381e06c4" containerName="istio-proxy"
Sep 18 16:25:14.962193 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.962011    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7d69023-7ac5-432b-982c-f4ef453e0103" containerName="istio-proxy"
Sep 18 16:25:14.964973 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.964952    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:14.970237 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.970206    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4f2d8a\"/\"e2e-conflict-a-5b8cb121-kserve-self-signed-certs\""
Sep 18 16:25:14.970380 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.970207    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4f2d8a\"/\"default-dockercfg-h4fvj\""
Sep 18 16:25:14.971785 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.971350    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4f2d8a\"/\"openshift-service-ca.crt\""
Sep 18 16:25:14.973091 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.973074    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4f2d8a\"/\"kube-root-ca.crt\""
Sep 18 16:25:14.995472 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:14.995446    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"]
Sep 18 16:25:15.098964 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.098930    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92f515ab-d334-4647-a667-9437252d1f71-tls-certs\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.099155 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.098971    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-home\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.099155 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.098990    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-model-cache\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.099155 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.099089    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-tmp-dir\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.099155 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.099117    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qqjj\" (UniqueName: \"kubernetes.io/projected/92f515ab-d334-4647-a667-9437252d1f71-kube-api-access-2qqjj\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.099331 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.099192    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-dshm\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.199829 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.199791    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-home\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.199829 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.199828    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-model-cache\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.200077 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.199881    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-tmp-dir\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.200077 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.199900    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qqjj\" (UniqueName: \"kubernetes.io/projected/92f515ab-d334-4647-a667-9437252d1f71-kube-api-access-2qqjj\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.200077 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.199947    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-dshm\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.200077 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.200026    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92f515ab-d334-4647-a667-9437252d1f71-tls-certs\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.200364 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.200212    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-home\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.200364 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.200242    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-model-cache\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.200364 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.200309    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-tmp-dir\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.202354 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.202331    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-dshm\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.202572 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.202552    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92f515ab-d334-4647-a667-9437252d1f71-tls-certs\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.213612 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.213555    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qqjj\" (UniqueName: \"kubernetes.io/projected/92f515ab-d334-4647-a667-9437252d1f71-kube-api-access-2qqjj\") pod \"e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.273855 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.273823    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:15.550193 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.550167    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"]
Sep 18 16:25:15.552253 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:25:15.552227    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92f515ab_d334_4647_a667_9437252d1f71.slice/crio-35ffe8e1d66fb3d27a2a780d639834bd34ac8bcdd8eaa18cbab16f3438138b66 WatchSource:0}: Error finding container 35ffe8e1d66fb3d27a2a780d639834bd34ac8bcdd8eaa18cbab16f3438138b66: Status 404 returned error can't find the container with id 35ffe8e1d66fb3d27a2a780d639834bd34ac8bcdd8eaa18cbab16f3438138b66
Sep 18 16:25:15.986419 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.986376    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j" event={"ID":"92f515ab-d334-4647-a667-9437252d1f71","Type":"ContainerStarted","Data":"c19fadda9b76259e4c2587edfd4ab6d665d61191300066985a3d46e92fc8cbd8"}
Sep 18 16:25:15.986823 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:15.986426    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j" event={"ID":"92f515ab-d334-4647-a667-9437252d1f71","Type":"ContainerStarted","Data":"35ffe8e1d66fb3d27a2a780d639834bd34ac8bcdd8eaa18cbab16f3438138b66"}
Sep 18 16:25:16.009963 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:16.009914    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j" podStartSLOduration=2.009897389 podStartE2EDuration="2.009897389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:25:16.007129038 +0000 UTC m=+1210.954627812" watchObservedRunningTime="2026-09-18 16:25:16.009897389 +0000 UTC m=+1210.957396189"
Sep 18 16:25:19.761738 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:19.761705    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"]
Sep 18 16:25:19.762136 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:19.762122    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52438627-318f-42f1-b2c1-1089381e06c4" containerName="istio-proxy"
Sep 18 16:25:19.762180 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:19.762138    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="52438627-318f-42f1-b2c1-1089381e06c4" containerName="istio-proxy"
Sep 18 16:25:19.764217 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:19.764199    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:19.768415 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:19.768394    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-5b98dc-openshift-default-dockercfg-dcq58\""
Sep 18 16:25:19.861711 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:19.861674    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"]
Sep 18 16:25:19.947208 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:19.947169    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-podinfo\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:19.947412 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:19.947217    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-data\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:19.947412 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:19.947317    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-token\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:19.947412 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:19.947365    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-workload-certs\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:19.947651 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:19.947426    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-workload-socket\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:19.947651 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:19.947451    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfc5l\" (UniqueName: \"kubernetes.io/projected/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-kube-api-access-vfc5l\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:19.947651 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:19.947495    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istiod-ca-cert\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:19.947651 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:19.947548    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-envoy\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:19.947651 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:19.947581    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-credential-socket\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.048968 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.048868    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-podinfo\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.048968 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.048927    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-data\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.048968 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.048969    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-token\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.049242 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.048998    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-workload-certs\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.049242 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.049038    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-workload-socket\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.049242 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.049055    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfc5l\" (UniqueName: \"kubernetes.io/projected/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-kube-api-access-vfc5l\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.049242 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.049081    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istiod-ca-cert\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.049242 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.049104    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-envoy\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.049242 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.049125    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-credential-socket\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.049560 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.049389    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-data\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.049560 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.049477    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-workload-certs\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.049669 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.049566    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-workload-socket\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.049749 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.049730    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-credential-socket\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.049853 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.049835    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istiod-ca-cert\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.051540 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.051495    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-envoy\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.051656 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.051590    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-podinfo\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.061645 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.061625    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-token\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.062003 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.061983    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfc5l\" (UniqueName: \"kubernetes.io/projected/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-kube-api-access-vfc5l\") pod \"e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") " pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.074633 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.074615    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:20.354740 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.354711    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"]
Sep 18 16:25:20.356567 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:25:20.356539    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb6fd0b_b70a_4e35_804f_030e67d4bf69.slice/crio-5bcb95906b5450da741e752ad7ee4d75f645c2931d151b3040caa11ccf57d14e WatchSource:0}: Error finding container 5bcb95906b5450da741e752ad7ee4d75f645c2931d151b3040caa11ccf57d14e: Status 404 returned error can't find the container with id 5bcb95906b5450da741e752ad7ee4d75f645c2931d151b3040caa11ccf57d14e
Sep 18 16:25:20.358734 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.358699    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:25:20.358802 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.358765    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:25:20.358802 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:20.358794    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:25:21.014074 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:21.014029    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z" event={"ID":"ecb6fd0b-b70a-4e35-804f-030e67d4bf69","Type":"ContainerStarted","Data":"4deba3560df69298d62c2368ddd72b87aa7da5f07f807225517b067c5f3bc696"}
Sep 18 16:25:21.014495 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:21.014091    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z" event={"ID":"ecb6fd0b-b70a-4e35-804f-030e67d4bf69","Type":"ContainerStarted","Data":"5bcb95906b5450da741e752ad7ee4d75f645c2931d151b3040caa11ccf57d14e"}
Sep 18 16:25:21.044879 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:21.044809    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z" podStartSLOduration=2.044787221 podStartE2EDuration="2.044787221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:25:21.043903555 +0000 UTC m=+1215.991402328" watchObservedRunningTime="2026-09-18 16:25:21.044787221 +0000 UTC m=+1215.992286008"
Sep 18 16:25:21.075823 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:21.075788    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:21.077429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:21.077403    2554 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 18 16:25:21.077575 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:21.077449    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z" podUID="ecb6fd0b-b70a-4e35-804f-030e67d4bf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 18 16:25:22.079694 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:22.079658    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:23.024407 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:23.024370    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:23.025329 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:23.025307    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:25:25.278123 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:25.278093    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:25.278505 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:25.278138    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:25.287134 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:25.287114    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:26.054040 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:26.054006    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:25:27.983855 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:27.983817    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"]
Sep 18 16:25:27.986474 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:27.986457    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:25:27.989768 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:27.989743    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-5b98dc-serving-cert\""
Sep 18 16:25:28.004271 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.004241    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"]
Sep 18 16:25:28.027950 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.027914    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6\" (UID: \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:25:28.028106 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.027961    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6\" (UID: \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:25:28.028106 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.028061    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47mhr\" (UniqueName: \"kubernetes.io/projected/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-kube-api-access-47mhr\") pod \"maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6\" (UID: \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:25:28.028106 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.028098    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6\" (UID: \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:25:28.128886 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.128852    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47mhr\" (UniqueName: \"kubernetes.io/projected/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-kube-api-access-47mhr\") pod \"maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6\" (UID: \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:25:28.129071 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.128904    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6\" (UID: \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:25:28.129071 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.129010    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6\" (UID: \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:25:28.129071 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.129055    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6\" (UID: \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:25:28.129735 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.129712    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6\" (UID: \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:25:28.131598 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.131576    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6\" (UID: \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:25:28.131684 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.131643    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6\" (UID: \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:25:28.140933 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.140914    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47mhr\" (UniqueName: \"kubernetes.io/projected/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-kube-api-access-47mhr\") pod \"maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6\" (UID: \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:25:28.297847 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.297761    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:25:28.584539 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.584433    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"]
Sep 18 16:25:28.587396 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:25:28.587370    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52502d4c_2ac8_4b53_8c2f_1c6ed2a367bb.slice/crio-0114183c643924ef2c3ff23f13433745c3759a32c313c7078963f4b338d0dd78 WatchSource:0}: Error finding container 0114183c643924ef2c3ff23f13433745c3759a32c313c7078963f4b338d0dd78: Status 404 returned error can't find the container with id 0114183c643924ef2c3ff23f13433745c3759a32c313c7078963f4b338d0dd78
Sep 18 16:25:28.800824 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.800789    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"]
Sep 18 16:25:28.803703 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.803685    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"
Sep 18 16:25:28.806509 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.806489    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-5b98dc\""
Sep 18 16:25:28.808196 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.808178    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-5b98dc-dockercfg-vlk6x\""
Sep 18 16:25:28.820967 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.820938    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"]
Sep 18 16:25:28.837123 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.837031    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ad33d7-6cf0-4bc7-ac01-423926cf2b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv\" (UID: \"60ad33d7-6cf0-4bc7-ac01-423926cf2b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"
Sep 18 16:25:28.837123 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.837073    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh6t5\" (UniqueName: \"kubernetes.io/projected/60ad33d7-6cf0-4bc7-ac01-423926cf2b1b-kube-api-access-lh6t5\") pod \"payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv\" (UID: \"60ad33d7-6cf0-4bc7-ac01-423926cf2b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"
Sep 18 16:25:28.938052 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.938015    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ad33d7-6cf0-4bc7-ac01-423926cf2b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv\" (UID: \"60ad33d7-6cf0-4bc7-ac01-423926cf2b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"
Sep 18 16:25:28.938052 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.938052    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh6t5\" (UniqueName: \"kubernetes.io/projected/60ad33d7-6cf0-4bc7-ac01-423926cf2b1b-kube-api-access-lh6t5\") pod \"payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv\" (UID: \"60ad33d7-6cf0-4bc7-ac01-423926cf2b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"
Sep 18 16:25:28.938774 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.938755    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ad33d7-6cf0-4bc7-ac01-423926cf2b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv\" (UID: \"60ad33d7-6cf0-4bc7-ac01-423926cf2b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"
Sep 18 16:25:28.947055 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:28.947033    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh6t5\" (UniqueName: \"kubernetes.io/projected/60ad33d7-6cf0-4bc7-ac01-423926cf2b1b-kube-api-access-lh6t5\") pod \"payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv\" (UID: \"60ad33d7-6cf0-4bc7-ac01-423926cf2b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"
Sep 18 16:25:29.059358 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:29.059323    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6" event={"ID":"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb","Type":"ContainerStarted","Data":"813813d66f8e55396c124f3780c0541ff600b4c04a2f0f80cb3532cc2d425fa3"}
Sep 18 16:25:29.059801 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:29.059378    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6" event={"ID":"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb","Type":"ContainerStarted","Data":"0114183c643924ef2c3ff23f13433745c3759a32c313c7078963f4b338d0dd78"}
Sep 18 16:25:29.059801 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:29.059490    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:25:29.085835 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:29.085781    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6" podStartSLOduration=1.7285477 podStartE2EDuration="2.085763983s" totalImagesPullingTime="357.216283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:25:29.084788319 +0000 UTC m=+1224.032287093" watchObservedRunningTime="2026-09-18 16:25:29.085763983 +0000 UTC m=+1224.033262756"
Sep 18 16:25:29.114666 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:29.114582    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"
Sep 18 16:25:29.394592 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:29.394487    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"]
Sep 18 16:25:29.397788 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:25:29.397756    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ad33d7_6cf0_4bc7_ac01_423926cf2b1b.slice/crio-9c3154846c4c8a63cf86f840872fca30847cd465a245d100a784864180d84630 WatchSource:0}: Error finding container 9c3154846c4c8a63cf86f840872fca30847cd465a245d100a784864180d84630: Status 404 returned error can't find the container with id 9c3154846c4c8a63cf86f840872fca30847cd465a245d100a784864180d84630
Sep 18 16:25:30.067928 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:30.067886    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv" event={"ID":"60ad33d7-6cf0-4bc7-ac01-423926cf2b1b","Type":"ContainerStarted","Data":"ded6034174fc1d4744284e101519885085cc689762ee8680b3d9e89bc937f7c0"}
Sep 18 16:25:30.068369 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:30.068122    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"
Sep 18 16:25:30.068369 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:30.068170    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv" event={"ID":"60ad33d7-6cf0-4bc7-ac01-423926cf2b1b","Type":"ContainerStarted","Data":"9c3154846c4c8a63cf86f840872fca30847cd465a245d100a784864180d84630"}
Sep 18 16:25:30.097979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:30.097920    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv" podStartSLOduration=2.09790215 podStartE2EDuration="2.09790215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:25:30.095831467 +0000 UTC m=+1225.043330261" watchObservedRunningTime="2026-09-18 16:25:30.09790215 +0000 UTC m=+1225.045400924"
Sep 18 16:25:35.073803 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:35.073770    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:25:41.076385 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:41.076343    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"
Sep 18 16:25:52.564621 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:52.561710    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"]
Sep 18 16:25:52.564621 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:52.562024    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz" podUID="f2fd3601-b37f-4446-a364-b1c1fd98cf4c" containerName="istio-proxy" containerID="cri-o://64e91f85e2f6c078f812efd31294292e1d89b58c7585081ca82ffff636296129" gracePeriod=30
Sep 18 16:25:53.799793 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.799742    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"]
Sep 18 16:25:53.802254 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.802235    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.805480 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.805450    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-5865ea-openshift-default-dockercfg-fxbr2\""
Sep 18 16:25:53.815510 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.815486    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"]
Sep 18 16:25:53.887678 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.887636    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-envoy\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.887869 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.887698    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-token\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.887869 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.887763    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-podinfo\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.887869 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.887783    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-credential-socket\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.887869 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.887814    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-data\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.887869 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.887831    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-workload-certs\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.888051 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.887903    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9953f9be-cbc3-44a4-a29d-4a301e468844-istiod-ca-cert\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.888051 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.887989    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-workload-socket\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.888051 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.888009    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj4bt\" (UniqueName: \"kubernetes.io/projected/9953f9be-cbc3-44a4-a29d-4a301e468844-kube-api-access-cj4bt\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.988883 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.988846    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-podinfo\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.988883 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.988885    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-credential-socket\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.989135 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.988913    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-data\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.989135 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.988931    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-workload-certs\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.989135 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.988958    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9953f9be-cbc3-44a4-a29d-4a301e468844-istiod-ca-cert\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.989135 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.989003    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-workload-socket\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.989135 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.989019    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj4bt\" (UniqueName: \"kubernetes.io/projected/9953f9be-cbc3-44a4-a29d-4a301e468844-kube-api-access-cj4bt\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.989135 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.989047    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-envoy\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.989135 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.989085    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-token\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.989500 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.989312    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-data\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.989500 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.989406    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-workload-certs\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.989646 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.989606    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-credential-socket\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.989714 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.989649    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-workload-socket\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.989836 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.989813    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9953f9be-cbc3-44a4-a29d-4a301e468844-istiod-ca-cert\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.991555 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.991512    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-envoy\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.991661 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.991643    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-podinfo\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.997715 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.997690    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-token\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:53.998085 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:53.998066    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj4bt\" (UniqueName: \"kubernetes.io/projected/9953f9be-cbc3-44a4-a29d-4a301e468844-kube-api-access-cj4bt\") pod \"e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") " pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:54.112694 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:54.112607    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:54.420128 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:54.420101    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"]
Sep 18 16:25:54.421860 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:25:54.421834    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9953f9be_cbc3_44a4_a29d_4a301e468844.slice/crio-6d558da3aa3211f483e38ca64a7bed77ddb4794f037540f81bb71ac40d51aa14 WatchSource:0}: Error finding container 6d558da3aa3211f483e38ca64a7bed77ddb4794f037540f81bb71ac40d51aa14: Status 404 returned error can't find the container with id 6d558da3aa3211f483e38ca64a7bed77ddb4794f037540f81bb71ac40d51aa14
Sep 18 16:25:54.424613 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:54.424581    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:25:54.424668 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:54.424646    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:25:54.424704 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:54.424675    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:25:55.204438 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:55.204343    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq" event={"ID":"9953f9be-cbc3-44a4-a29d-4a301e468844","Type":"ContainerStarted","Data":"9922900e3a276845c17395b8124e886e7da75853f80028d14e9e42929cf473ab"}
Sep 18 16:25:55.204810 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:55.204449    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq" event={"ID":"9953f9be-cbc3-44a4-a29d-4a301e468844","Type":"ContainerStarted","Data":"6d558da3aa3211f483e38ca64a7bed77ddb4794f037540f81bb71ac40d51aa14"}
Sep 18 16:25:55.232620 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:55.232569    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq" podStartSLOduration=2.232551534 podStartE2EDuration="2.232551534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:25:55.231497287 +0000 UTC m=+1250.178996064" watchObservedRunningTime="2026-09-18 16:25:55.232551534 +0000 UTC m=+1250.180050308"
Sep 18 16:25:56.112931 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:56.112899    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:56.118975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:56.118945    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:56.210123 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:56.210089    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:56.211411 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:56.211392    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:25:56.882142 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:56.882104    2554 patch_prober.go:28] interesting pod/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 16:25:56.882319 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:56.882178    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz" podUID="f2fd3601-b37f-4446-a364-b1c1fd98cf4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:25:57.818868 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.818842    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:25:57.930695 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.930670    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pfgz\" (UniqueName: \"kubernetes.io/projected/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-kube-api-access-9pfgz\") pod \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") "
Sep 18 16:25:57.930859 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.930729    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-credential-socket\") pod \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") "
Sep 18 16:25:57.930859 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.930746    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-envoy\") pod \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") "
Sep 18 16:25:57.930859 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.930798    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istiod-ca-cert\") pod \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") "
Sep 18 16:25:57.930859 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.930826    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-podinfo\") pod \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") "
Sep 18 16:25:57.930859 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.930842    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-token\") pod \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") "
Sep 18 16:25:57.931124 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.930880    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-data\") pod \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") "
Sep 18 16:25:57.931124 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.930912    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-workload-certs\") pod \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") "
Sep 18 16:25:57.931124 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.930945    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2fd3601-b37f-4446-a364-b1c1fd98cf4c" (UID: "f2fd3601-b37f-4446-a364-b1c1fd98cf4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:25:57.931124 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.930978    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-workload-socket\") pod \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\" (UID: \"f2fd3601-b37f-4446-a364-b1c1fd98cf4c\") "
Sep 18 16:25:57.931328 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.931120    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2fd3601-b37f-4446-a364-b1c1fd98cf4c" (UID: "f2fd3601-b37f-4446-a364-b1c1fd98cf4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:25:57.931328 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.931152    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2fd3601-b37f-4446-a364-b1c1fd98cf4c" (UID: "f2fd3601-b37f-4446-a364-b1c1fd98cf4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:25:57.931328 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.931274    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2fd3601-b37f-4446-a364-b1c1fd98cf4c" (UID: "f2fd3601-b37f-4446-a364-b1c1fd98cf4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:25:57.931328 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.931307    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2fd3601-b37f-4446-a364-b1c1fd98cf4c" (UID: "f2fd3601-b37f-4446-a364-b1c1fd98cf4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:25:57.931673 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.931651    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istiod-ca-cert\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:57.931789 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.931683    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-data\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:57.931789 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.931696    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-workload-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:57.931789 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.931707    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-workload-socket\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:57.931789 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.931719    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-credential-socket\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:57.933331 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.933307    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2fd3601-b37f-4446-a364-b1c1fd98cf4c" (UID: "f2fd3601-b37f-4446-a364-b1c1fd98cf4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:25:57.933424 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.933324    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2fd3601-b37f-4446-a364-b1c1fd98cf4c" (UID: "f2fd3601-b37f-4446-a364-b1c1fd98cf4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:25:57.933599 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.933581    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-kube-api-access-9pfgz" (OuterVolumeSpecName: "kube-api-access-9pfgz") pod "f2fd3601-b37f-4446-a364-b1c1fd98cf4c" (UID: "f2fd3601-b37f-4446-a364-b1c1fd98cf4c"). InnerVolumeSpecName "kube-api-access-9pfgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:25:57.933648 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:57.933584    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2fd3601-b37f-4446-a364-b1c1fd98cf4c" (UID: "f2fd3601-b37f-4446-a364-b1c1fd98cf4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:25:58.032761 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.032730    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pfgz\" (UniqueName: \"kubernetes.io/projected/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-kube-api-access-9pfgz\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:58.045071 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.032768    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-envoy\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:58.045071 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.032783    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-podinfo\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:58.045071 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.032794    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fd3601-b37f-4446-a364-b1c1fd98cf4c-istio-token\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:25:58.221879 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.221800    2554 generic.go:356] "Generic (PLEG): container finished" podID="ecb6fd0b-b70a-4e35-804f-030e67d4bf69" containerID="4deba3560df69298d62c2368ddd72b87aa7da5f07f807225517b067c5f3bc696" exitCode=0
Sep 18 16:25:58.222057 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.221879    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z" event={"ID":"ecb6fd0b-b70a-4e35-804f-030e67d4bf69","Type":"ContainerDied","Data":"4deba3560df69298d62c2368ddd72b87aa7da5f07f807225517b067c5f3bc696"}
Sep 18 16:25:58.222383 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.222358    2554 scope.go:117] "RemoveContainer" containerID="4deba3560df69298d62c2368ddd72b87aa7da5f07f807225517b067c5f3bc696"
Sep 18 16:25:58.223240 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.223222    2554 generic.go:356] "Generic (PLEG): container finished" podID="f2fd3601-b37f-4446-a364-b1c1fd98cf4c" containerID="64e91f85e2f6c078f812efd31294292e1d89b58c7585081ca82ffff636296129" exitCode=0
Sep 18 16:25:58.223343 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.223325    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"
Sep 18 16:25:58.223416 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.223323    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz" event={"ID":"f2fd3601-b37f-4446-a364-b1c1fd98cf4c","Type":"ContainerDied","Data":"64e91f85e2f6c078f812efd31294292e1d89b58c7585081ca82ffff636296129"}
Sep 18 16:25:58.223416 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.223390    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz" event={"ID":"f2fd3601-b37f-4446-a364-b1c1fd98cf4c","Type":"ContainerDied","Data":"3a298a269bae5e4331eaea3a7772f2f03e8339a71a68f60e0246d96402fd13aa"}
Sep 18 16:25:58.223554 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.223421    2554 scope.go:117] "RemoveContainer" containerID="64e91f85e2f6c078f812efd31294292e1d89b58c7585081ca82ffff636296129"
Sep 18 16:25:58.223630 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.223598    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:25:58.223700 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.223683    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:25:58.223767 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.223728    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:25:58.236072 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.236048    2554 scope.go:117] "RemoveContainer" containerID="64e91f85e2f6c078f812efd31294292e1d89b58c7585081ca82ffff636296129"
Sep 18 16:25:58.236388 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:25:58.236367    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64e91f85e2f6c078f812efd31294292e1d89b58c7585081ca82ffff636296129\": container with ID starting with 64e91f85e2f6c078f812efd31294292e1d89b58c7585081ca82ffff636296129 not found: ID does not exist" containerID="64e91f85e2f6c078f812efd31294292e1d89b58c7585081ca82ffff636296129"
Sep 18 16:25:58.236440 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.236402    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64e91f85e2f6c078f812efd31294292e1d89b58c7585081ca82ffff636296129"} err="failed to get container status \"64e91f85e2f6c078f812efd31294292e1d89b58c7585081ca82ffff636296129\": rpc error: code = NotFound desc = could not find container \"64e91f85e2f6c078f812efd31294292e1d89b58c7585081ca82ffff636296129\": container with ID starting with 64e91f85e2f6c078f812efd31294292e1d89b58c7585081ca82ffff636296129 not found: ID does not exist"
Sep 18 16:25:58.318137 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.318103    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"]
Sep 18 16:25:58.334904 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:58.334871    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-72547f-openshift-default-6ff497ddfd-4zsnz"]
Sep 18 16:25:59.230317 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:59.230277    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z" event={"ID":"ecb6fd0b-b70a-4e35-804f-030e67d4bf69","Type":"ContainerStarted","Data":"ff38be622c4fd7fac3c92e23de18d629f31e40a99a054083ef668b7bd2979bb8"}
Sep 18 16:25:59.277865 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:25:59.277820    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2fd3601-b37f-4446-a364-b1c1fd98cf4c" path="/var/lib/kubelet/pods/f2fd3601-b37f-4446-a364-b1c1fd98cf4c/volumes"
Sep 18 16:26:00.074824 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:00.074791    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:26:00.075004 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:00.074963    2554 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 18 16:26:00.075049 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:00.075011    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z" podUID="ecb6fd0b-b70a-4e35-804f-030e67d4bf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 18 16:26:01.075373 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:01.075328    2554 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 18 16:26:01.075850 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:01.075399    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z" podUID="ecb6fd0b-b70a-4e35-804f-030e67d4bf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 18 16:26:02.075150 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:02.075115    2554 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 18 16:26:02.075338 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:02.075172    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z" podUID="ecb6fd0b-b70a-4e35-804f-030e67d4bf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 18 16:26:03.075000 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:03.074965    2554 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 18 16:26:03.075495 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:03.075030    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z" podUID="ecb6fd0b-b70a-4e35-804f-030e67d4bf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 18 16:26:03.663149 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:03.663109    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"]
Sep 18 16:26:03.663565 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:03.663551    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fd3601-b37f-4446-a364-b1c1fd98cf4c" containerName="istio-proxy"
Sep 18 16:26:03.663629 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:03.663566    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fd3601-b37f-4446-a364-b1c1fd98cf4c" containerName="istio-proxy"
Sep 18 16:26:03.663674 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:03.663669    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2fd3601-b37f-4446-a364-b1c1fd98cf4c" containerName="istio-proxy"
Sep 18 16:26:03.666699 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:03.666683    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"
Sep 18 16:26:03.669773 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:03.669748    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-5865ea\""
Sep 18 16:26:03.681086 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:03.681052    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjjkl\" (UniqueName: \"kubernetes.io/projected/58dbe8aa-7537-4391-8486-981345f7c0b5-kube-api-access-fjjkl\") pod \"payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk\" (UID: \"58dbe8aa-7537-4391-8486-981345f7c0b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"
Sep 18 16:26:03.681209 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:03.681108    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58dbe8aa-7537-4391-8486-981345f7c0b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk\" (UID: \"58dbe8aa-7537-4391-8486-981345f7c0b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"
Sep 18 16:26:03.682924 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:03.682900    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"]
Sep 18 16:26:03.781740 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:03.781698    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58dbe8aa-7537-4391-8486-981345f7c0b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk\" (UID: \"58dbe8aa-7537-4391-8486-981345f7c0b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"
Sep 18 16:26:03.781924 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:03.781859    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjjkl\" (UniqueName: \"kubernetes.io/projected/58dbe8aa-7537-4391-8486-981345f7c0b5-kube-api-access-fjjkl\") pod \"payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk\" (UID: \"58dbe8aa-7537-4391-8486-981345f7c0b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"
Sep 18 16:26:03.782386 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:03.782363    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58dbe8aa-7537-4391-8486-981345f7c0b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk\" (UID: \"58dbe8aa-7537-4391-8486-981345f7c0b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"
Sep 18 16:26:03.792137 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:03.792113    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjjkl\" (UniqueName: \"kubernetes.io/projected/58dbe8aa-7537-4391-8486-981345f7c0b5-kube-api-access-fjjkl\") pod \"payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk\" (UID: \"58dbe8aa-7537-4391-8486-981345f7c0b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"
Sep 18 16:26:03.977388 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:03.977358    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"
Sep 18 16:26:04.075772 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:04.075249    2554 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 18 16:26:04.075772 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:04.075313    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z" podUID="ecb6fd0b-b70a-4e35-804f-030e67d4bf69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 18 16:26:04.278473 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:04.278445    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"]
Sep 18 16:26:04.279938 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:26:04.279906    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58dbe8aa_7537_4391_8486_981345f7c0b5.slice/crio-f1c875d24df5547c788afcdbc08282b6a433e94a683a2a4a5b1b06b5d8ffaecd WatchSource:0}: Error finding container f1c875d24df5547c788afcdbc08282b6a433e94a683a2a4a5b1b06b5d8ffaecd: Status 404 returned error can't find the container with id f1c875d24df5547c788afcdbc08282b6a433e94a683a2a4a5b1b06b5d8ffaecd
Sep 18 16:26:05.079492 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:05.079461    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:26:05.080000 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:05.079847    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:26:05.080437 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:05.080416    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:26:05.278464 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:05.278435    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk" event={"ID":"58dbe8aa-7537-4391-8486-981345f7c0b5","Type":"ContainerStarted","Data":"d1bfff5be14a53d7c2c17331d5bb79ed00ccb9bf0eb17012038f39c93118707b"}
Sep 18 16:26:05.278648 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:05.278481    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk" event={"ID":"58dbe8aa-7537-4391-8486-981345f7c0b5","Type":"ContainerStarted","Data":"f1c875d24df5547c788afcdbc08282b6a433e94a683a2a4a5b1b06b5d8ffaecd"}
Sep 18 16:26:05.291473 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:05.291423    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk" podStartSLOduration=2.29140962 podStartE2EDuration="2.29140962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:05.290379364 +0000 UTC m=+1260.237878141" watchObservedRunningTime="2026-09-18 16:26:05.29140962 +0000 UTC m=+1260.238908436"
Sep 18 16:26:06.278661 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:06.278617    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"
Sep 18 16:26:17.284672 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:17.284644    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"
Sep 18 16:26:26.211709 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:26.211668    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 18 16:26:26.212060 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:26.211759    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq" podUID="9953f9be-cbc3-44a4-a29d-4a301e468844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 18 16:26:26.392201 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:26.392126    2554 generic.go:356] "Generic (PLEG): container finished" podID="9953f9be-cbc3-44a4-a29d-4a301e468844" containerID="9922900e3a276845c17395b8124e886e7da75853f80028d14e9e42929cf473ab" exitCode=0
Sep 18 16:26:26.392339 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:26.392197    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq" event={"ID":"9953f9be-cbc3-44a4-a29d-4a301e468844","Type":"ContainerDied","Data":"9922900e3a276845c17395b8124e886e7da75853f80028d14e9e42929cf473ab"}
Sep 18 16:26:26.392636 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:26.392622    2554 scope.go:117] "RemoveContainer" containerID="9922900e3a276845c17395b8124e886e7da75853f80028d14e9e42929cf473ab"
Sep 18 16:26:26.393881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:26.393853    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:26:26.393943 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:26.393919    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:26:26.393979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:26.393949    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:26:27.399597 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:27.399554    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq" event={"ID":"9953f9be-cbc3-44a4-a29d-4a301e468844","Type":"ContainerStarted","Data":"3bc2c66dbda8ff393b938c13578d72d02282d4e3f88302a68da4e4be640e3fc5"}
Sep 18 16:26:28.113733 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:28.113701    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:26:28.113922 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:28.113870    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 18 16:26:28.113970 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:28.113916    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq" podUID="9953f9be-cbc3-44a4-a29d-4a301e468844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 18 16:26:29.113705 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:29.113671    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 18 16:26:29.114192 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:29.113741    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq" podUID="9953f9be-cbc3-44a4-a29d-4a301e468844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 18 16:26:30.113865 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:30.113826    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 18 16:26:30.114361 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:30.113908    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq" podUID="9953f9be-cbc3-44a4-a29d-4a301e468844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 18 16:26:31.113802 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:31.113767    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 18 16:26:31.113967 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:31.113817    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq" podUID="9953f9be-cbc3-44a4-a29d-4a301e468844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 18 16:26:32.113395 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:32.113360    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 18 16:26:32.113603 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:32.113417    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq" podUID="9953f9be-cbc3-44a4-a29d-4a301e468844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 18 16:26:33.117513 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:33.117480    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:26:33.117992 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:33.117873    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:26:33.118549 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:33.118506    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:26:36.868475 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:36.868441    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"]
Sep 18 16:26:36.871425 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:36.871409    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:36.874035 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:36.874014    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-157a333a-openshift-default-dockercfg-2jtlb\""
Sep 18 16:26:36.893314 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:36.893292    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"]
Sep 18 16:26:36.922339 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:36.922313    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-workload-socket\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:36.922506 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:36.922348    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-podinfo\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:36.922506 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:36.922379    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0c52af-c871-41d1-b1be-d7e6d0359421-istiod-ca-cert\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:36.922506 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:36.922464    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-data\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:36.922725 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:36.922581    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-workload-certs\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:36.922725 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:36.922636    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-envoy\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:36.922725 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:36.922679    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-token\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:36.922828 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:36.922747    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqvcc\" (UniqueName: \"kubernetes.io/projected/1a0c52af-c871-41d1-b1be-d7e6d0359421-kube-api-access-wqvcc\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:36.922828 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:36.922797    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-credential-socket\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.023772 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.023726    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-credential-socket\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.023974 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.023794    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-workload-socket\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.023974 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.023826    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-podinfo\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.023974 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.023854    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0c52af-c871-41d1-b1be-d7e6d0359421-istiod-ca-cert\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.023974 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.023892    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-data\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.023974 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.023944    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-workload-certs\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.024260 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.023987    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-envoy\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.024260 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.024027    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-token\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.024260 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.024077    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqvcc\" (UniqueName: \"kubernetes.io/projected/1a0c52af-c871-41d1-b1be-d7e6d0359421-kube-api-access-wqvcc\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.024260 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.024191    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-workload-socket\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.024473 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.024380    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-data\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.024473 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.024404    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-workload-certs\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.024613 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.024501    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-credential-socket\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.024661 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.024612    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0c52af-c871-41d1-b1be-d7e6d0359421-istiod-ca-cert\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.026403 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.026382    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-envoy\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.026734 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.026712    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-podinfo\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.034027 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.034004    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-token\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.034684 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.034666    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqvcc\" (UniqueName: \"kubernetes.io/projected/1a0c52af-c871-41d1-b1be-d7e6d0359421-kube-api-access-wqvcc\") pod \"e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") " pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.181673 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.181647    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:37.480107 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.480080    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"]
Sep 18 16:26:37.481628 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:26:37.481601    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0c52af_c871_41d1_b1be_d7e6d0359421.slice/crio-673de939ab47836d7395f35b12c9740152fcf048ee43500a3b00a1e09b4d38af WatchSource:0}: Error finding container 673de939ab47836d7395f35b12c9740152fcf048ee43500a3b00a1e09b4d38af: Status 404 returned error can't find the container with id 673de939ab47836d7395f35b12c9740152fcf048ee43500a3b00a1e09b4d38af
Sep 18 16:26:37.483927 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.483890    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:26:37.484007 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.483957    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:26:37.484007 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:37.483987    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:26:38.457907 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:38.457872    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p" event={"ID":"1a0c52af-c871-41d1-b1be-d7e6d0359421","Type":"ContainerStarted","Data":"78532b4f0a1d39a6cb549df9a4bbf1f994ce14d2056016a9ed9670e8136ec842"}
Sep 18 16:26:38.458308 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:38.457935    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p" event={"ID":"1a0c52af-c871-41d1-b1be-d7e6d0359421","Type":"ContainerStarted","Data":"673de939ab47836d7395f35b12c9740152fcf048ee43500a3b00a1e09b4d38af"}
Sep 18 16:26:38.488403 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:38.488359    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p" podStartSLOduration=2.488344705 podStartE2EDuration="2.488344705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:38.487961567 +0000 UTC m=+1293.435460340" watchObservedRunningTime="2026-09-18 16:26:38.488344705 +0000 UTC m=+1293.435843478"
Sep 18 16:26:39.182824 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:39.182787    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:39.187763 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:39.187729    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:39.462799 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:39.462714    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:39.463779 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:39.463753    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:26:48.929259 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:48.929229    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"]
Sep 18 16:26:48.931927 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:48.931905    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:26:48.941588 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:48.941557    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-157a333a-serving-cert\""
Sep 18 16:26:48.963313 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:48.963287    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"]
Sep 18 16:26:49.040178 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.040141    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aecc371-576d-4608-a04a-bb2979283dd6-openshift-service-ca\") pod \"maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn\" (UID: \"4aecc371-576d-4608-a04a-bb2979283dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:26:49.040363 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.040188    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aecc371-576d-4608-a04a-bb2979283dd6-metrics-certs\") pod \"maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn\" (UID: \"4aecc371-576d-4608-a04a-bb2979283dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:26:49.040363 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.040219    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg8pp\" (UniqueName: \"kubernetes.io/projected/4aecc371-576d-4608-a04a-bb2979283dd6-kube-api-access-hg8pp\") pod \"maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn\" (UID: \"4aecc371-576d-4608-a04a-bb2979283dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:26:49.040363 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.040351    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aecc371-576d-4608-a04a-bb2979283dd6-maas-api-tls\") pod \"maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn\" (UID: \"4aecc371-576d-4608-a04a-bb2979283dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:26:49.141511 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.141475    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aecc371-576d-4608-a04a-bb2979283dd6-openshift-service-ca\") pod \"maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn\" (UID: \"4aecc371-576d-4608-a04a-bb2979283dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:26:49.141511 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.141515    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aecc371-576d-4608-a04a-bb2979283dd6-metrics-certs\") pod \"maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn\" (UID: \"4aecc371-576d-4608-a04a-bb2979283dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:26:49.141747 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.141560    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg8pp\" (UniqueName: \"kubernetes.io/projected/4aecc371-576d-4608-a04a-bb2979283dd6-kube-api-access-hg8pp\") pod \"maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn\" (UID: \"4aecc371-576d-4608-a04a-bb2979283dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:26:49.141747 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.141643    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aecc371-576d-4608-a04a-bb2979283dd6-maas-api-tls\") pod \"maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn\" (UID: \"4aecc371-576d-4608-a04a-bb2979283dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:26:49.142317 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.142236    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aecc371-576d-4608-a04a-bb2979283dd6-openshift-service-ca\") pod \"maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn\" (UID: \"4aecc371-576d-4608-a04a-bb2979283dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:26:49.144195 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.144163    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aecc371-576d-4608-a04a-bb2979283dd6-metrics-certs\") pod \"maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn\" (UID: \"4aecc371-576d-4608-a04a-bb2979283dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:26:49.144280 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.144197    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aecc371-576d-4608-a04a-bb2979283dd6-maas-api-tls\") pod \"maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn\" (UID: \"4aecc371-576d-4608-a04a-bb2979283dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:26:49.154142 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.154116    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg8pp\" (UniqueName: \"kubernetes.io/projected/4aecc371-576d-4608-a04a-bb2979283dd6-kube-api-access-hg8pp\") pod \"maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn\" (UID: \"4aecc371-576d-4608-a04a-bb2979283dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:26:49.242999 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.242924    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:26:49.526764 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.526739    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"]
Sep 18 16:26:49.528385 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:26:49.528362    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aecc371_576d_4608_a04a_bb2979283dd6.slice/crio-1668d4977f1809fee367dccde53cce8beb288eed1218b1a22bb6f08952e3f411 WatchSource:0}: Error finding container 1668d4977f1809fee367dccde53cce8beb288eed1218b1a22bb6f08952e3f411: Status 404 returned error can't find the container with id 1668d4977f1809fee367dccde53cce8beb288eed1218b1a22bb6f08952e3f411
Sep 18 16:26:49.529774 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.529756    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:26:49.896816 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.896737    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"]
Sep 18 16:26:49.900147 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.900130    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"
Sep 18 16:26:49.903143 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.903117    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-157a333a\""
Sep 18 16:26:49.903270 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.903194    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-157a333a-dockercfg-vq24t\""
Sep 18 16:26:49.911398 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.911375    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"]
Sep 18 16:26:49.950609 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.950567    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292e7e8e-4fc2-45d7-958f-8f21f3283137-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7\" (UID: \"292e7e8e-4fc2-45d7-958f-8f21f3283137\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"
Sep 18 16:26:49.951071 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:49.950677    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d96qj\" (UniqueName: \"kubernetes.io/projected/292e7e8e-4fc2-45d7-958f-8f21f3283137-kube-api-access-d96qj\") pod \"payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7\" (UID: \"292e7e8e-4fc2-45d7-958f-8f21f3283137\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"
Sep 18 16:26:50.052305 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:50.052274    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292e7e8e-4fc2-45d7-958f-8f21f3283137-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7\" (UID: \"292e7e8e-4fc2-45d7-958f-8f21f3283137\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"
Sep 18 16:26:50.052438 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:50.052320    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d96qj\" (UniqueName: \"kubernetes.io/projected/292e7e8e-4fc2-45d7-958f-8f21f3283137-kube-api-access-d96qj\") pod \"payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7\" (UID: \"292e7e8e-4fc2-45d7-958f-8f21f3283137\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"
Sep 18 16:26:50.052943 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:50.052924    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292e7e8e-4fc2-45d7-958f-8f21f3283137-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7\" (UID: \"292e7e8e-4fc2-45d7-958f-8f21f3283137\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"
Sep 18 16:26:50.063411 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:50.063391    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d96qj\" (UniqueName: \"kubernetes.io/projected/292e7e8e-4fc2-45d7-958f-8f21f3283137-kube-api-access-d96qj\") pod \"payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7\" (UID: \"292e7e8e-4fc2-45d7-958f-8f21f3283137\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"
Sep 18 16:26:50.212388 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:50.212352    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"
Sep 18 16:26:50.488807 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:50.488779    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"]
Sep 18 16:26:50.490899 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:26:50.490875    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292e7e8e_4fc2_45d7_958f_8f21f3283137.slice/crio-a534a8918754ea913b390478a16b20c8c41df95793a63b958958bbe08d930ed3 WatchSource:0}: Error finding container a534a8918754ea913b390478a16b20c8c41df95793a63b958958bbe08d930ed3: Status 404 returned error can't find the container with id a534a8918754ea913b390478a16b20c8c41df95793a63b958958bbe08d930ed3
Sep 18 16:26:50.521202 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:50.521168    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn" event={"ID":"4aecc371-576d-4608-a04a-bb2979283dd6","Type":"ContainerStarted","Data":"a83b39330903227f1d1da6e91da1b1109948f255838e1151421ca0baae5e4679"}
Sep 18 16:26:50.521324 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:50.521233    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:26:50.521324 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:50.521254    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn" event={"ID":"4aecc371-576d-4608-a04a-bb2979283dd6","Type":"ContainerStarted","Data":"1668d4977f1809fee367dccde53cce8beb288eed1218b1a22bb6f08952e3f411"}
Sep 18 16:26:50.522385 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:50.522365    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7" event={"ID":"292e7e8e-4fc2-45d7-958f-8f21f3283137","Type":"ContainerStarted","Data":"a534a8918754ea913b390478a16b20c8c41df95793a63b958958bbe08d930ed3"}
Sep 18 16:26:50.550567 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:50.550505    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn" podStartSLOduration=2.116316323 podStartE2EDuration="2.550491347s" totalImagesPullingTime="434.175024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:50.548572999 +0000 UTC m=+1305.496071792" watchObservedRunningTime="2026-09-18 16:26:50.550491347 +0000 UTC m=+1305.497990120"
Sep 18 16:26:51.529506 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:51.529466    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7" event={"ID":"292e7e8e-4fc2-45d7-958f-8f21f3283137","Type":"ContainerStarted","Data":"7dee75819eae8315bbc38e01abb8d70255f9315c72c85f8a97c1050dee67fad7"}
Sep 18 16:26:51.557343 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:51.557285    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7" podStartSLOduration=2.557271465 podStartE2EDuration="2.557271465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:51.554985227 +0000 UTC m=+1306.502484015" watchObservedRunningTime="2026-09-18 16:26:51.557271465 +0000 UTC m=+1306.504770239"
Sep 18 16:26:52.535188 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:52.535149    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"
Sep 18 16:26:56.535703 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:56.535677    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:26:57.020608 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.020570    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"]
Sep 18 16:26:57.020802 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.020773    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw" podUID="380b3a73-8056-4f47-9005-00e48e7a0b32" containerName="maas-api" containerID="cri-o://83609f04ddfa1dc1c8b996c02344f5b980e20fee90e5016ee5bbdf447c501654" gracePeriod=30
Sep 18 16:26:57.278462 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.278402    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:26:57.329808 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.329777    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380b3a73-8056-4f47-9005-00e48e7a0b32-metrics-certs\") pod \"380b3a73-8056-4f47-9005-00e48e7a0b32\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") "
Sep 18 16:26:57.329808 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.329818    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380b3a73-8056-4f47-9005-00e48e7a0b32-maas-api-tls\") pod \"380b3a73-8056-4f47-9005-00e48e7a0b32\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") "
Sep 18 16:26:57.330063 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.329906    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380b3a73-8056-4f47-9005-00e48e7a0b32-openshift-service-ca\") pod \"380b3a73-8056-4f47-9005-00e48e7a0b32\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") "
Sep 18 16:26:57.330152 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.330131    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msw5t\" (UniqueName: \"kubernetes.io/projected/380b3a73-8056-4f47-9005-00e48e7a0b32-kube-api-access-msw5t\") pod \"380b3a73-8056-4f47-9005-00e48e7a0b32\" (UID: \"380b3a73-8056-4f47-9005-00e48e7a0b32\") "
Sep 18 16:26:57.330994 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.330924    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380b3a73-8056-4f47-9005-00e48e7a0b32-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "380b3a73-8056-4f47-9005-00e48e7a0b32" (UID: "380b3a73-8056-4f47-9005-00e48e7a0b32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:26:57.333700 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.333670    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380b3a73-8056-4f47-9005-00e48e7a0b32-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "380b3a73-8056-4f47-9005-00e48e7a0b32" (UID: "380b3a73-8056-4f47-9005-00e48e7a0b32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:26:57.334001 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.333940    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380b3a73-8056-4f47-9005-00e48e7a0b32-kube-api-access-msw5t" (OuterVolumeSpecName: "kube-api-access-msw5t") pod "380b3a73-8056-4f47-9005-00e48e7a0b32" (UID: "380b3a73-8056-4f47-9005-00e48e7a0b32"). InnerVolumeSpecName "kube-api-access-msw5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:26:57.334222 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.334198    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380b3a73-8056-4f47-9005-00e48e7a0b32-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "380b3a73-8056-4f47-9005-00e48e7a0b32" (UID: "380b3a73-8056-4f47-9005-00e48e7a0b32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:26:57.431783 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.431743    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380b3a73-8056-4f47-9005-00e48e7a0b32-metrics-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:26:57.432020 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.431998    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380b3a73-8056-4f47-9005-00e48e7a0b32-maas-api-tls\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:26:57.432020 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.432021    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380b3a73-8056-4f47-9005-00e48e7a0b32-openshift-service-ca\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:26:57.432020 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.432037    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msw5t\" (UniqueName: \"kubernetes.io/projected/380b3a73-8056-4f47-9005-00e48e7a0b32-kube-api-access-msw5t\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:26:57.563712 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.563605    2554 generic.go:356] "Generic (PLEG): container finished" podID="380b3a73-8056-4f47-9005-00e48e7a0b32" containerID="83609f04ddfa1dc1c8b996c02344f5b980e20fee90e5016ee5bbdf447c501654" exitCode=0
Sep 18 16:26:57.563712 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.563687    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"
Sep 18 16:26:57.564259 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.563739    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw" event={"ID":"380b3a73-8056-4f47-9005-00e48e7a0b32","Type":"ContainerDied","Data":"83609f04ddfa1dc1c8b996c02344f5b980e20fee90e5016ee5bbdf447c501654"}
Sep 18 16:26:57.564259 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.563794    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw" event={"ID":"380b3a73-8056-4f47-9005-00e48e7a0b32","Type":"ContainerDied","Data":"86baeae8149a8b25ad6ab7b6d51c27065bf87bda457dcd1492d5ee2c3f422aa6"}
Sep 18 16:26:57.564259 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.563817    2554 scope.go:117] "RemoveContainer" containerID="83609f04ddfa1dc1c8b996c02344f5b980e20fee90e5016ee5bbdf447c501654"
Sep 18 16:26:57.581160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.581008    2554 scope.go:117] "RemoveContainer" containerID="83609f04ddfa1dc1c8b996c02344f5b980e20fee90e5016ee5bbdf447c501654"
Sep 18 16:26:57.581300 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:26:57.581262    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83609f04ddfa1dc1c8b996c02344f5b980e20fee90e5016ee5bbdf447c501654\": container with ID starting with 83609f04ddfa1dc1c8b996c02344f5b980e20fee90e5016ee5bbdf447c501654 not found: ID does not exist" containerID="83609f04ddfa1dc1c8b996c02344f5b980e20fee90e5016ee5bbdf447c501654"
Sep 18 16:26:57.581426 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.581300    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83609f04ddfa1dc1c8b996c02344f5b980e20fee90e5016ee5bbdf447c501654"} err="failed to get container status \"83609f04ddfa1dc1c8b996c02344f5b980e20fee90e5016ee5bbdf447c501654\": rpc error: code = NotFound desc = could not find container \"83609f04ddfa1dc1c8b996c02344f5b980e20fee90e5016ee5bbdf447c501654\": container with ID starting with 83609f04ddfa1dc1c8b996c02344f5b980e20fee90e5016ee5bbdf447c501654 not found: ID does not exist"
Sep 18 16:26:57.611897 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.611867    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"]
Sep 18 16:26:57.616876 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:57.616853    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fbc529-6d944cd4c8-7hssw"]
Sep 18 16:26:59.278072 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:26:59.278040    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="380b3a73-8056-4f47-9005-00e48e7a0b32" path="/var/lib/kubelet/pods/380b3a73-8056-4f47-9005-00e48e7a0b32/volumes"
Sep 18 16:27:03.542758 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:03.542722    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"
Sep 18 16:27:09.464709 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:09.464682    2554 patch_prober.go:28] interesting pod/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 18 16:27:09.465098 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:09.464731    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p" podUID="1a0c52af-c871-41d1-b1be-d7e6d0359421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 18 16:27:09.629457 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:09.629421    2554 generic.go:356] "Generic (PLEG): container finished" podID="1a0c52af-c871-41d1-b1be-d7e6d0359421" containerID="78532b4f0a1d39a6cb549df9a4bbf1f994ce14d2056016a9ed9670e8136ec842" exitCode=0
Sep 18 16:27:09.629648 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:09.629463    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p" event={"ID":"1a0c52af-c871-41d1-b1be-d7e6d0359421","Type":"ContainerDied","Data":"78532b4f0a1d39a6cb549df9a4bbf1f994ce14d2056016a9ed9670e8136ec842"}
Sep 18 16:27:09.629879 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:09.629864    2554 scope.go:117] "RemoveContainer" containerID="78532b4f0a1d39a6cb549df9a4bbf1f994ce14d2056016a9ed9670e8136ec842"
Sep 18 16:27:09.631072 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:09.631043    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:27:09.631132 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:09.631112    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:27:09.631168 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:09.631142    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:27:10.640603 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:10.639656    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p" event={"ID":"1a0c52af-c871-41d1-b1be-d7e6d0359421","Type":"ContainerStarted","Data":"5d91c72a962ceecee30ea790e819a6ff52fff2bb65913c258e16770d1bfd90ce"}
Sep 18 16:27:11.182306 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:11.182264    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:27:11.182568 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:11.182544    2554 patch_prober.go:28] interesting pod/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 18 16:27:11.182629 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:11.182600    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p" podUID="1a0c52af-c871-41d1-b1be-d7e6d0359421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 18 16:27:12.182147 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:12.182105    2554 patch_prober.go:28] interesting pod/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 18 16:27:12.182560 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:12.182185    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p" podUID="1a0c52af-c871-41d1-b1be-d7e6d0359421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 18 16:27:13.182276 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:13.182235    2554 patch_prober.go:28] interesting pod/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 18 16:27:13.182718 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:13.182314    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p" podUID="1a0c52af-c871-41d1-b1be-d7e6d0359421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 18 16:27:14.182913 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:14.182872    2554 patch_prober.go:28] interesting pod/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 18 16:27:14.183392 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:14.182983    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p" podUID="1a0c52af-c871-41d1-b1be-d7e6d0359421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 18 16:27:15.182958 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:15.182923    2554 patch_prober.go:28] interesting pod/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 18 16:27:15.183347 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:15.182993    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p" podUID="1a0c52af-c871-41d1-b1be-d7e6d0359421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 18 16:27:16.187044 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:16.187013    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:27:16.187427 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:16.187385    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:27:16.188034 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:16.188016    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:27:16.760666 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:16.760632    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"]
Sep 18 16:27:16.760904 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:16.760879    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z" podUID="76fea163-46ab-4e45-9b9b-f62073e880ee" containerName="maas-api" containerID="cri-o://59333b9687dc0527535a5b6f01297effc457be0526333cd86cd064cb55c6abe8" gracePeriod=30
Sep 18 16:27:17.019367 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.019308    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:27:17.037762 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.037739    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76fea163-46ab-4e45-9b9b-f62073e880ee-openshift-service-ca\") pod \"76fea163-46ab-4e45-9b9b-f62073e880ee\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") "
Sep 18 16:27:17.037877 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.037768    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-metrics-certs\") pod \"76fea163-46ab-4e45-9b9b-f62073e880ee\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") "
Sep 18 16:27:17.037877 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.037786    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7chj\" (UniqueName: \"kubernetes.io/projected/76fea163-46ab-4e45-9b9b-f62073e880ee-kube-api-access-n7chj\") pod \"76fea163-46ab-4e45-9b9b-f62073e880ee\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") "
Sep 18 16:27:17.038158 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.038122    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fea163-46ab-4e45-9b9b-f62073e880ee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "76fea163-46ab-4e45-9b9b-f62073e880ee" (UID: "76fea163-46ab-4e45-9b9b-f62073e880ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:27:17.039947 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.039924    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "76fea163-46ab-4e45-9b9b-f62073e880ee" (UID: "76fea163-46ab-4e45-9b9b-f62073e880ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:27:17.040109 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.040086    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fea163-46ab-4e45-9b9b-f62073e880ee-kube-api-access-n7chj" (OuterVolumeSpecName: "kube-api-access-n7chj") pod "76fea163-46ab-4e45-9b9b-f62073e880ee" (UID: "76fea163-46ab-4e45-9b9b-f62073e880ee"). InnerVolumeSpecName "kube-api-access-n7chj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:27:17.138810 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.138777    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-maas-api-tls\") pod \"76fea163-46ab-4e45-9b9b-f62073e880ee\" (UID: \"76fea163-46ab-4e45-9b9b-f62073e880ee\") "
Sep 18 16:27:17.139094 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.139075    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76fea163-46ab-4e45-9b9b-f62073e880ee-openshift-service-ca\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:27:17.139166 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.139099    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-metrics-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:27:17.139166 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.139112    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7chj\" (UniqueName: \"kubernetes.io/projected/76fea163-46ab-4e45-9b9b-f62073e880ee-kube-api-access-n7chj\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:27:17.140940 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.140920    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "76fea163-46ab-4e45-9b9b-f62073e880ee" (UID: "76fea163-46ab-4e45-9b9b-f62073e880ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:27:17.239975 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.239933    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76fea163-46ab-4e45-9b9b-f62073e880ee-maas-api-tls\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:27:17.685644 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.685608    2554 generic.go:356] "Generic (PLEG): container finished" podID="76fea163-46ab-4e45-9b9b-f62073e880ee" containerID="59333b9687dc0527535a5b6f01297effc457be0526333cd86cd064cb55c6abe8" exitCode=0
Sep 18 16:27:17.685858 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.685651    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"
Sep 18 16:27:17.685858 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.685696    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z" event={"ID":"76fea163-46ab-4e45-9b9b-f62073e880ee","Type":"ContainerDied","Data":"59333b9687dc0527535a5b6f01297effc457be0526333cd86cd064cb55c6abe8"}
Sep 18 16:27:17.685858 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.685754    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z" event={"ID":"76fea163-46ab-4e45-9b9b-f62073e880ee","Type":"ContainerDied","Data":"f8ce5fa2b77004bbfdaa9d26625950c3ff9a9c0ddde9019a1ad4d4d3180dfb27"}
Sep 18 16:27:17.685858 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.685784    2554 scope.go:117] "RemoveContainer" containerID="59333b9687dc0527535a5b6f01297effc457be0526333cd86cd064cb55c6abe8"
Sep 18 16:27:17.696042 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.696026    2554 scope.go:117] "RemoveContainer" containerID="59333b9687dc0527535a5b6f01297effc457be0526333cd86cd064cb55c6abe8"
Sep 18 16:27:17.696309 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:27:17.696290    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59333b9687dc0527535a5b6f01297effc457be0526333cd86cd064cb55c6abe8\": container with ID starting with 59333b9687dc0527535a5b6f01297effc457be0526333cd86cd064cb55c6abe8 not found: ID does not exist" containerID="59333b9687dc0527535a5b6f01297effc457be0526333cd86cd064cb55c6abe8"
Sep 18 16:27:17.696384 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.696316    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59333b9687dc0527535a5b6f01297effc457be0526333cd86cd064cb55c6abe8"} err="failed to get container status \"59333b9687dc0527535a5b6f01297effc457be0526333cd86cd064cb55c6abe8\": rpc error: code = NotFound desc = could not find container \"59333b9687dc0527535a5b6f01297effc457be0526333cd86cd064cb55c6abe8\": container with ID starting with 59333b9687dc0527535a5b6f01297effc457be0526333cd86cd064cb55c6abe8 not found: ID does not exist"
Sep 18 16:27:17.711849 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.711822    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"]
Sep 18 16:27:17.718605 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:17.718582    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f22789-795f6b8676-8cb6z"]
Sep 18 16:27:19.277345 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:27:19.277311    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76fea163-46ab-4e45-9b9b-f62073e880ee" path="/var/lib/kubelet/pods/76fea163-46ab-4e45-9b9b-f62073e880ee/volumes"
Sep 18 16:28:12.446516 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.446478    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r"]
Sep 18 16:28:12.447019 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.447004    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fea163-46ab-4e45-9b9b-f62073e880ee" containerName="maas-api"
Sep 18 16:28:12.447066 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.447021    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fea163-46ab-4e45-9b9b-f62073e880ee" containerName="maas-api"
Sep 18 16:28:12.447066 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.447038    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380b3a73-8056-4f47-9005-00e48e7a0b32" containerName="maas-api"
Sep 18 16:28:12.447066 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.447043    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="380b3a73-8056-4f47-9005-00e48e7a0b32" containerName="maas-api"
Sep 18 16:28:12.447163 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.447116    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="380b3a73-8056-4f47-9005-00e48e7a0b32" containerName="maas-api"
Sep 18 16:28:12.447163 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.447126    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="76fea163-46ab-4e45-9b9b-f62073e880ee" containerName="maas-api"
Sep 18 16:28:12.449750 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.449734    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r"
Sep 18 16:28:12.455005 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.454981    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-a75117-dockercfg-lqq85\""
Sep 18 16:28:12.455202 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.455045    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-a75117\""
Sep 18 16:28:12.475865 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.475839    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r"]
Sep 18 16:28:12.563184 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.563155    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc4f966-9449-43ce-85e9-f024c59d9b8b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r\" (UID: \"3bc4f966-9449-43ce-85e9-f024c59d9b8b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r"
Sep 18 16:28:12.563353 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.563318    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpvn7\" (UniqueName: \"kubernetes.io/projected/3bc4f966-9449-43ce-85e9-f024c59d9b8b-kube-api-access-wpvn7\") pod \"payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r\" (UID: \"3bc4f966-9449-43ce-85e9-f024c59d9b8b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r"
Sep 18 16:28:12.664371 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.664340    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpvn7\" (UniqueName: \"kubernetes.io/projected/3bc4f966-9449-43ce-85e9-f024c59d9b8b-kube-api-access-wpvn7\") pod \"payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r\" (UID: \"3bc4f966-9449-43ce-85e9-f024c59d9b8b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r"
Sep 18 16:28:12.664579 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.664389    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc4f966-9449-43ce-85e9-f024c59d9b8b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r\" (UID: \"3bc4f966-9449-43ce-85e9-f024c59d9b8b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r"
Sep 18 16:28:12.665079 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.665059    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc4f966-9449-43ce-85e9-f024c59d9b8b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r\" (UID: \"3bc4f966-9449-43ce-85e9-f024c59d9b8b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r"
Sep 18 16:28:12.675457 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.675430    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpvn7\" (UniqueName: \"kubernetes.io/projected/3bc4f966-9449-43ce-85e9-f024c59d9b8b-kube-api-access-wpvn7\") pod \"payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r\" (UID: \"3bc4f966-9449-43ce-85e9-f024c59d9b8b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r"
Sep 18 16:28:12.760113 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:12.760015    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r"
Sep 18 16:28:13.038813 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:13.038741    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r"]
Sep 18 16:28:13.041877 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:28:13.041851    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc4f966_9449_43ce_85e9_f024c59d9b8b.slice/crio-e7f27b7107907b9a59d17513f14d9e9e4a1578822c92d5ca0ec7970a26a9c245 WatchSource:0}: Error finding container e7f27b7107907b9a59d17513f14d9e9e4a1578822c92d5ca0ec7970a26a9c245: Status 404 returned error can't find the container with id e7f27b7107907b9a59d17513f14d9e9e4a1578822c92d5ca0ec7970a26a9c245
Sep 18 16:28:13.984611 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:13.984562    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r" event={"ID":"3bc4f966-9449-43ce-85e9-f024c59d9b8b","Type":"ContainerStarted","Data":"7fe50cf43f0f8ba5f4becae79f3f3d9cece66e12c9ff462b9111e370d71f3a2f"}
Sep 18 16:28:13.984989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:13.984664    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r"
Sep 18 16:28:13.984989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:13.984689    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r" event={"ID":"3bc4f966-9449-43ce-85e9-f024c59d9b8b","Type":"ContainerStarted","Data":"e7f27b7107907b9a59d17513f14d9e9e4a1578822c92d5ca0ec7970a26a9c245"}
Sep 18 16:28:14.011082 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:14.011036    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r" podStartSLOduration=2.011023098 podStartE2EDuration="2.011023098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:28:14.008484682 +0000 UTC m=+1388.955983469" watchObservedRunningTime="2026-09-18 16:28:14.011023098 +0000 UTC m=+1388.958521873"
Sep 18 16:28:24.992140 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:24.992112    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r"
Sep 18 16:28:41.708576 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:41.708541    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"]
Sep 18 16:28:41.711048 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:41.711027    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:28:41.713433 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:41.713414    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-11725e-serving-cert\""
Sep 18 16:28:41.729343 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:41.729323    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"]
Sep 18 16:28:41.748350 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:41.748323    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60bb6aff-82be-45fb-abdf-d7daec90c255-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s\" (UID: \"60bb6aff-82be-45fb-abdf-d7daec90c255\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:28:41.748452 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:41.748435    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60bb6aff-82be-45fb-abdf-d7daec90c255-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s\" (UID: \"60bb6aff-82be-45fb-abdf-d7daec90c255\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:28:41.748540 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:41.748505    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hdvr\" (UniqueName: \"kubernetes.io/projected/60bb6aff-82be-45fb-abdf-d7daec90c255-kube-api-access-2hdvr\") pod \"maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s\" (UID: \"60bb6aff-82be-45fb-abdf-d7daec90c255\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:28:41.748628 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:41.748607    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60bb6aff-82be-45fb-abdf-d7daec90c255-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s\" (UID: \"60bb6aff-82be-45fb-abdf-d7daec90c255\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:28:41.849824 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:41.849793    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60bb6aff-82be-45fb-abdf-d7daec90c255-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s\" (UID: \"60bb6aff-82be-45fb-abdf-d7daec90c255\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:28:41.849996 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:41.849853    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hdvr\" (UniqueName: \"kubernetes.io/projected/60bb6aff-82be-45fb-abdf-d7daec90c255-kube-api-access-2hdvr\") pod \"maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s\" (UID: \"60bb6aff-82be-45fb-abdf-d7daec90c255\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:28:41.849996 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:41.849890    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60bb6aff-82be-45fb-abdf-d7daec90c255-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s\" (UID: \"60bb6aff-82be-45fb-abdf-d7daec90c255\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:28:41.849996 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:41.849932    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60bb6aff-82be-45fb-abdf-d7daec90c255-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s\" (UID: \"60bb6aff-82be-45fb-abdf-d7daec90c255\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:28:41.850107 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:28:41.850035    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-serving-cert: secret "maas-api-e2e-shared-b-w3-11725e-serving-cert" not found
Sep 18 16:28:41.850142 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:28:41.850112    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60bb6aff-82be-45fb-abdf-d7daec90c255-maas-api-tls podName:60bb6aff-82be-45fb-abdf-d7daec90c255 nodeName:}" failed. No retries permitted until 2026-09-18 16:28:42.350091598 +0000 UTC m=+1417.297590367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60bb6aff-82be-45fb-abdf-d7daec90c255-maas-api-tls") pod "maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s" (UID: "60bb6aff-82be-45fb-abdf-d7daec90c255") : secret "maas-api-e2e-shared-b-w3-11725e-serving-cert" not found
Sep 18 16:28:41.850646 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:41.850623    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60bb6aff-82be-45fb-abdf-d7daec90c255-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s\" (UID: \"60bb6aff-82be-45fb-abdf-d7daec90c255\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:28:41.852409 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:41.852382    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60bb6aff-82be-45fb-abdf-d7daec90c255-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s\" (UID: \"60bb6aff-82be-45fb-abdf-d7daec90c255\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:28:41.862793 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:41.862776    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hdvr\" (UniqueName: \"kubernetes.io/projected/60bb6aff-82be-45fb-abdf-d7daec90c255-kube-api-access-2hdvr\") pod \"maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s\" (UID: \"60bb6aff-82be-45fb-abdf-d7daec90c255\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:28:42.354767 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:42.354733    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60bb6aff-82be-45fb-abdf-d7daec90c255-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s\" (UID: \"60bb6aff-82be-45fb-abdf-d7daec90c255\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:28:42.357194 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:42.357173    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60bb6aff-82be-45fb-abdf-d7daec90c255-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s\" (UID: \"60bb6aff-82be-45fb-abdf-d7daec90c255\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:28:42.622217 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:42.622120    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:28:42.900901 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:42.900874    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"]
Sep 18 16:28:42.902756 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:28:42.902722    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60bb6aff_82be_45fb_abdf_d7daec90c255.slice/crio-e82046047556458234a5cceb0d293633755aa51845ed8c9c408db511b6417b74 WatchSource:0}: Error finding container e82046047556458234a5cceb0d293633755aa51845ed8c9c408db511b6417b74: Status 404 returned error can't find the container with id e82046047556458234a5cceb0d293633755aa51845ed8c9c408db511b6417b74
Sep 18 16:28:43.145739 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:43.145702    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s" event={"ID":"60bb6aff-82be-45fb-abdf-d7daec90c255","Type":"ContainerStarted","Data":"e82046047556458234a5cceb0d293633755aa51845ed8c9c408db511b6417b74"}
Sep 18 16:28:44.152049 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:44.152007    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s" event={"ID":"60bb6aff-82be-45fb-abdf-d7daec90c255","Type":"ContainerStarted","Data":"ebd580cfceb3eb80b39d5046c5df682d621b2914ce4c80b7dd8d5ded966881ba"}
Sep 18 16:28:44.152444 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:44.152079    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:28:44.181500 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:44.181455    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s" podStartSLOduration=2.824371255 podStartE2EDuration="3.181440757s" totalImagesPullingTime="357.069502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:28:44.180877039 +0000 UTC m=+1419.128375824" watchObservedRunningTime="2026-09-18 16:28:44.181440757 +0000 UTC m=+1419.128939531"
Sep 18 16:28:50.163575 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:28:50.163475    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s"
Sep 18 16:29:11.726139 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:11.726102    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"]
Sep 18 16:29:26.746220 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:26.746152    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j" podUID="92f515ab-d334-4647-a667-9437252d1f71" containerName="main" containerID="cri-o://c19fadda9b76259e4c2587edfd4ab6d665d61191300066985a3d46e92fc8cbd8" gracePeriod=45
Sep 18 16:29:26.995388 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:26.995365    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:29:27.086354 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.086290    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-tmp-dir\") pod \"92f515ab-d334-4647-a667-9437252d1f71\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") "
Sep 18 16:29:27.086354 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.086333    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-home\") pod \"92f515ab-d334-4647-a667-9437252d1f71\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") "
Sep 18 16:29:27.086354 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.086357    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92f515ab-d334-4647-a667-9437252d1f71-tls-certs\") pod \"92f515ab-d334-4647-a667-9437252d1f71\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") "
Sep 18 16:29:27.086638 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.086373    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-dshm\") pod \"92f515ab-d334-4647-a667-9437252d1f71\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") "
Sep 18 16:29:27.086638 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.086416    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-model-cache\") pod \"92f515ab-d334-4647-a667-9437252d1f71\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") "
Sep 18 16:29:27.086638 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.086476    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qqjj\" (UniqueName: \"kubernetes.io/projected/92f515ab-d334-4647-a667-9437252d1f71-kube-api-access-2qqjj\") pod \"92f515ab-d334-4647-a667-9437252d1f71\" (UID: \"92f515ab-d334-4647-a667-9437252d1f71\") "
Sep 18 16:29:27.086638 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.086626    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-home" (OuterVolumeSpecName: "home") pod "92f515ab-d334-4647-a667-9437252d1f71" (UID: "92f515ab-d334-4647-a667-9437252d1f71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:29:27.086845 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.086641    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92f515ab-d334-4647-a667-9437252d1f71" (UID: "92f515ab-d334-4647-a667-9437252d1f71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:29:27.086845 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.086740    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-model-cache" (OuterVolumeSpecName: "model-cache") pod "92f515ab-d334-4647-a667-9437252d1f71" (UID: "92f515ab-d334-4647-a667-9437252d1f71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:29:27.086950 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.086906    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-model-cache\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:29:27.086950 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.086921    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-tmp-dir\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:29:27.086950 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.086930    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-home\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:29:27.088779 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.088749    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-dshm" (OuterVolumeSpecName: "dshm") pod "92f515ab-d334-4647-a667-9437252d1f71" (UID: "92f515ab-d334-4647-a667-9437252d1f71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:29:27.088881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.088810    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92f515ab-d334-4647-a667-9437252d1f71-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92f515ab-d334-4647-a667-9437252d1f71" (UID: "92f515ab-d334-4647-a667-9437252d1f71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:29:27.088881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.088824    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f515ab-d334-4647-a667-9437252d1f71-kube-api-access-2qqjj" (OuterVolumeSpecName: "kube-api-access-2qqjj") pod "92f515ab-d334-4647-a667-9437252d1f71" (UID: "92f515ab-d334-4647-a667-9437252d1f71"). InnerVolumeSpecName "kube-api-access-2qqjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:29:27.187935 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.187904    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qqjj\" (UniqueName: \"kubernetes.io/projected/92f515ab-d334-4647-a667-9437252d1f71-kube-api-access-2qqjj\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:29:27.187935 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.187932    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92f515ab-d334-4647-a667-9437252d1f71-tls-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:29:27.187935 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.187942    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92f515ab-d334-4647-a667-9437252d1f71-dshm\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:29:27.379209 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.379100    2554 generic.go:356] "Generic (PLEG): container finished" podID="92f515ab-d334-4647-a667-9437252d1f71" containerID="c19fadda9b76259e4c2587edfd4ab6d665d61191300066985a3d46e92fc8cbd8" exitCode=0
Sep 18 16:29:27.379209 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.379140    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"
Sep 18 16:29:27.379209 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.379187    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j" event={"ID":"92f515ab-d334-4647-a667-9437252d1f71","Type":"ContainerDied","Data":"c19fadda9b76259e4c2587edfd4ab6d665d61191300066985a3d46e92fc8cbd8"}
Sep 18 16:29:27.379510 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.379231    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j" event={"ID":"92f515ab-d334-4647-a667-9437252d1f71","Type":"ContainerDied","Data":"35ffe8e1d66fb3d27a2a780d639834bd34ac8bcdd8eaa18cbab16f3438138b66"}
Sep 18 16:29:27.379510 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.379249    2554 scope.go:117] "RemoveContainer" containerID="c19fadda9b76259e4c2587edfd4ab6d665d61191300066985a3d46e92fc8cbd8"
Sep 18 16:29:27.392683 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.392653    2554 scope.go:117] "RemoveContainer" containerID="c19fadda9b76259e4c2587edfd4ab6d665d61191300066985a3d46e92fc8cbd8"
Sep 18 16:29:27.393007 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:29:27.392986    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c19fadda9b76259e4c2587edfd4ab6d665d61191300066985a3d46e92fc8cbd8\": container with ID starting with c19fadda9b76259e4c2587edfd4ab6d665d61191300066985a3d46e92fc8cbd8 not found: ID does not exist" containerID="c19fadda9b76259e4c2587edfd4ab6d665d61191300066985a3d46e92fc8cbd8"
Sep 18 16:29:27.393060 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:27.393022    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c19fadda9b76259e4c2587edfd4ab6d665d61191300066985a3d46e92fc8cbd8"} err="failed to get container status \"c19fadda9b76259e4c2587edfd4ab6d665d61191300066985a3d46e92fc8cbd8\": rpc error: code = NotFound desc = could not find container \"c19fadda9b76259e4c2587edfd4ab6d665d61191300066985a3d46e92fc8cbd8\": container with ID starting with c19fadda9b76259e4c2587edfd4ab6d665d61191300066985a3d46e92fc8cbd8 not found: ID does not exist"
Sep 18 16:29:29.930028 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:29.929996    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"]
Sep 18 16:29:30.831553 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:30.831499    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"]
Sep 18 16:29:30.831872 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:30.831841    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn" podUID="4aecc371-576d-4608-a04a-bb2979283dd6" containerName="maas-api" containerID="cri-o://a83b39330903227f1d1da6e91da1b1109948f255838e1151421ca0baae5e4679" gracePeriod=30
Sep 18 16:29:30.833583 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:30.833561    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-a-5b8cb121-kserve-56cf9f4b7-hxw5j"]
Sep 18 16:29:31.076707 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.076682    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:29:31.127621 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.127558    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aecc371-576d-4608-a04a-bb2979283dd6-metrics-certs\") pod \"4aecc371-576d-4608-a04a-bb2979283dd6\" (UID: \"4aecc371-576d-4608-a04a-bb2979283dd6\") "
Sep 18 16:29:31.127760 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.127622    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aecc371-576d-4608-a04a-bb2979283dd6-openshift-service-ca\") pod \"4aecc371-576d-4608-a04a-bb2979283dd6\" (UID: \"4aecc371-576d-4608-a04a-bb2979283dd6\") "
Sep 18 16:29:31.127760 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.127651    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg8pp\" (UniqueName: \"kubernetes.io/projected/4aecc371-576d-4608-a04a-bb2979283dd6-kube-api-access-hg8pp\") pod \"4aecc371-576d-4608-a04a-bb2979283dd6\" (UID: \"4aecc371-576d-4608-a04a-bb2979283dd6\") "
Sep 18 16:29:31.127760 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.127698    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aecc371-576d-4608-a04a-bb2979283dd6-maas-api-tls\") pod \"4aecc371-576d-4608-a04a-bb2979283dd6\" (UID: \"4aecc371-576d-4608-a04a-bb2979283dd6\") "
Sep 18 16:29:31.127983 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.127961    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aecc371-576d-4608-a04a-bb2979283dd6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4aecc371-576d-4608-a04a-bb2979283dd6" (UID: "4aecc371-576d-4608-a04a-bb2979283dd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:29:31.129905 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.129880    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aecc371-576d-4608-a04a-bb2979283dd6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4aecc371-576d-4608-a04a-bb2979283dd6" (UID: "4aecc371-576d-4608-a04a-bb2979283dd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:29:31.129905 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.129895    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aecc371-576d-4608-a04a-bb2979283dd6-kube-api-access-hg8pp" (OuterVolumeSpecName: "kube-api-access-hg8pp") pod "4aecc371-576d-4608-a04a-bb2979283dd6" (UID: "4aecc371-576d-4608-a04a-bb2979283dd6"). InnerVolumeSpecName "kube-api-access-hg8pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:29:31.130035 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.129900    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aecc371-576d-4608-a04a-bb2979283dd6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4aecc371-576d-4608-a04a-bb2979283dd6" (UID: "4aecc371-576d-4608-a04a-bb2979283dd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:29:31.229147 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.229118    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aecc371-576d-4608-a04a-bb2979283dd6-maas-api-tls\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:29:31.229147 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.229142    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aecc371-576d-4608-a04a-bb2979283dd6-metrics-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:29:31.229286 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.229151    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aecc371-576d-4608-a04a-bb2979283dd6-openshift-service-ca\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:29:31.229286 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.229160    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg8pp\" (UniqueName: \"kubernetes.io/projected/4aecc371-576d-4608-a04a-bb2979283dd6-kube-api-access-hg8pp\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:29:31.278381 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.278350    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92f515ab-d334-4647-a667-9437252d1f71" path="/var/lib/kubelet/pods/92f515ab-d334-4647-a667-9437252d1f71/volumes"
Sep 18 16:29:31.401996 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.401918    2554 generic.go:356] "Generic (PLEG): container finished" podID="4aecc371-576d-4608-a04a-bb2979283dd6" containerID="a83b39330903227f1d1da6e91da1b1109948f255838e1151421ca0baae5e4679" exitCode=0
Sep 18 16:29:31.401996 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.401971    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"
Sep 18 16:29:31.402163 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.402009    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn" event={"ID":"4aecc371-576d-4608-a04a-bb2979283dd6","Type":"ContainerDied","Data":"a83b39330903227f1d1da6e91da1b1109948f255838e1151421ca0baae5e4679"}
Sep 18 16:29:31.402163 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.402064    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn" event={"ID":"4aecc371-576d-4608-a04a-bb2979283dd6","Type":"ContainerDied","Data":"1668d4977f1809fee367dccde53cce8beb288eed1218b1a22bb6f08952e3f411"}
Sep 18 16:29:31.402163 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.402096    2554 scope.go:117] "RemoveContainer" containerID="a83b39330903227f1d1da6e91da1b1109948f255838e1151421ca0baae5e4679"
Sep 18 16:29:31.411151 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.411134    2554 scope.go:117] "RemoveContainer" containerID="a83b39330903227f1d1da6e91da1b1109948f255838e1151421ca0baae5e4679"
Sep 18 16:29:31.411423 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:29:31.411402    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83b39330903227f1d1da6e91da1b1109948f255838e1151421ca0baae5e4679\": container with ID starting with a83b39330903227f1d1da6e91da1b1109948f255838e1151421ca0baae5e4679 not found: ID does not exist" containerID="a83b39330903227f1d1da6e91da1b1109948f255838e1151421ca0baae5e4679"
Sep 18 16:29:31.411469 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:31.411432    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83b39330903227f1d1da6e91da1b1109948f255838e1151421ca0baae5e4679"} err="failed to get container status \"a83b39330903227f1d1da6e91da1b1109948f255838e1151421ca0baae5e4679\": rpc error: code = NotFound desc = could not find container \"a83b39330903227f1d1da6e91da1b1109948f255838e1151421ca0baae5e4679\": container with ID starting with a83b39330903227f1d1da6e91da1b1109948f255838e1151421ca0baae5e4679 not found: ID does not exist"
Sep 18 16:29:33.147114 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:33.147080    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"]
Sep 18 16:29:34.310131 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:34.310098    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-157a333a-6b9675cdf5-7rxrn"]
Sep 18 16:29:35.277833 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:35.277801    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aecc371-576d-4608-a04a-bb2979283dd6" path="/var/lib/kubelet/pods/4aecc371-576d-4608-a04a-bb2979283dd6/volumes"
Sep 18 16:29:36.902813 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:36.902778    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"]
Sep 18 16:29:36.903246 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:36.903051    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7" podUID="292e7e8e-4fc2-45d7-958f-8f21f3283137" containerName="payload-pre-processing" containerID="cri-o://7dee75819eae8315bbc38e01abb8d70255f9315c72c85f8a97c1050dee67fad7" gracePeriod=30
Sep 18 16:29:37.171183 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:37.171152    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"
Sep 18 16:29:37.287900 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:37.287869    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292e7e8e-4fc2-45d7-958f-8f21f3283137-plugins-config-volume\") pod \"292e7e8e-4fc2-45d7-958f-8f21f3283137\" (UID: \"292e7e8e-4fc2-45d7-958f-8f21f3283137\") "
Sep 18 16:29:37.288051 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:37.287990    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d96qj\" (UniqueName: \"kubernetes.io/projected/292e7e8e-4fc2-45d7-958f-8f21f3283137-kube-api-access-d96qj\") pod \"292e7e8e-4fc2-45d7-958f-8f21f3283137\" (UID: \"292e7e8e-4fc2-45d7-958f-8f21f3283137\") "
Sep 18 16:29:37.288211 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:37.288190    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292e7e8e-4fc2-45d7-958f-8f21f3283137-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "292e7e8e-4fc2-45d7-958f-8f21f3283137" (UID: "292e7e8e-4fc2-45d7-958f-8f21f3283137"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:29:37.290117 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:37.290090    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292e7e8e-4fc2-45d7-958f-8f21f3283137-kube-api-access-d96qj" (OuterVolumeSpecName: "kube-api-access-d96qj") pod "292e7e8e-4fc2-45d7-958f-8f21f3283137" (UID: "292e7e8e-4fc2-45d7-958f-8f21f3283137"). InnerVolumeSpecName "kube-api-access-d96qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:29:37.389305 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:37.389269    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292e7e8e-4fc2-45d7-958f-8f21f3283137-plugins-config-volume\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:29:37.389305 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:37.389298    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d96qj\" (UniqueName: \"kubernetes.io/projected/292e7e8e-4fc2-45d7-958f-8f21f3283137-kube-api-access-d96qj\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:29:37.437306 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:37.437272    2554 generic.go:356] "Generic (PLEG): container finished" podID="292e7e8e-4fc2-45d7-958f-8f21f3283137" containerID="7dee75819eae8315bbc38e01abb8d70255f9315c72c85f8a97c1050dee67fad7" exitCode=0
Sep 18 16:29:37.437549 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:37.437317    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"
Sep 18 16:29:37.437549 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:37.437324    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7" event={"ID":"292e7e8e-4fc2-45d7-958f-8f21f3283137","Type":"ContainerDied","Data":"7dee75819eae8315bbc38e01abb8d70255f9315c72c85f8a97c1050dee67fad7"}
Sep 18 16:29:37.437549 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:37.437375    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7" event={"ID":"292e7e8e-4fc2-45d7-958f-8f21f3283137","Type":"ContainerDied","Data":"a534a8918754ea913b390478a16b20c8c41df95793a63b958958bbe08d930ed3"}
Sep 18 16:29:37.437549 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:37.437404    2554 scope.go:117] "RemoveContainer" containerID="7dee75819eae8315bbc38e01abb8d70255f9315c72c85f8a97c1050dee67fad7"
Sep 18 16:29:37.446977 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:37.446958    2554 scope.go:117] "RemoveContainer" containerID="7dee75819eae8315bbc38e01abb8d70255f9315c72c85f8a97c1050dee67fad7"
Sep 18 16:29:37.447294 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:29:37.447270    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dee75819eae8315bbc38e01abb8d70255f9315c72c85f8a97c1050dee67fad7\": container with ID starting with 7dee75819eae8315bbc38e01abb8d70255f9315c72c85f8a97c1050dee67fad7 not found: ID does not exist" containerID="7dee75819eae8315bbc38e01abb8d70255f9315c72c85f8a97c1050dee67fad7"
Sep 18 16:29:37.447369 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:37.447303    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dee75819eae8315bbc38e01abb8d70255f9315c72c85f8a97c1050dee67fad7"} err="failed to get container status \"7dee75819eae8315bbc38e01abb8d70255f9315c72c85f8a97c1050dee67fad7\": rpc error: code = NotFound desc = could not find container \"7dee75819eae8315bbc38e01abb8d70255f9315c72c85f8a97c1050dee67fad7\": container with ID starting with 7dee75819eae8315bbc38e01abb8d70255f9315c72c85f8a97c1050dee67fad7 not found: ID does not exist"
Sep 18 16:29:37.479937 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:37.479903    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"]
Sep 18 16:29:37.491903 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:37.491875    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-157a333a-7c6956fd5d-zhws7"]
Sep 18 16:29:39.277873 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:39.277839    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="292e7e8e-4fc2-45d7-958f-8f21f3283137" path="/var/lib/kubelet/pods/292e7e8e-4fc2-45d7-958f-8f21f3283137/volumes"
Sep 18 16:29:58.855954 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.855915    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"]
Sep 18 16:29:58.856434 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.856340    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aecc371-576d-4608-a04a-bb2979283dd6" containerName="maas-api"
Sep 18 16:29:58.856434 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.856351    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aecc371-576d-4608-a04a-bb2979283dd6" containerName="maas-api"
Sep 18 16:29:58.856434 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.856368    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292e7e8e-4fc2-45d7-958f-8f21f3283137" containerName="payload-pre-processing"
Sep 18 16:29:58.856434 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.856373    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="292e7e8e-4fc2-45d7-958f-8f21f3283137" containerName="payload-pre-processing"
Sep 18 16:29:58.856434 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.856388    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92f515ab-d334-4647-a667-9437252d1f71" containerName="main"
Sep 18 16:29:58.856434 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.856393    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="92f515ab-d334-4647-a667-9437252d1f71" containerName="main"
Sep 18 16:29:58.856700 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.856463    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aecc371-576d-4608-a04a-bb2979283dd6" containerName="maas-api"
Sep 18 16:29:58.856700 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.856472    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="92f515ab-d334-4647-a667-9437252d1f71" containerName="main"
Sep 18 16:29:58.856700 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.856479    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="292e7e8e-4fc2-45d7-958f-8f21f3283137" containerName="payload-pre-processing"
Sep 18 16:29:58.859672 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.859654    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.862471 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.862450    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a75117\"/\"openshift-service-ca.crt\""
Sep 18 16:29:58.863231 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.863208    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a75117\"/\"kube-root-ca.crt\""
Sep 18 16:29:58.863347 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.863246    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a75117\"/\"default-dockercfg-jshkt\""
Sep 18 16:29:58.863555 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.863536    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a75117\"/\"auth-test-model-a75117-kserve-self-signed-certs\""
Sep 18 16:29:58.873171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.873149    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"]
Sep 18 16:29:58.894584 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.894559    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk8lv\" (UniqueName: \"kubernetes.io/projected/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-kube-api-access-lk8lv\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.894682 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.894614    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-home\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.894725 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.894698    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-model-cache\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.894766 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.894753    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-dshm\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.894800 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.894780    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-tls-certs\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.894877 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.894860    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-tmp-dir\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.995820 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.995791    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-home\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.995986 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.995836    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-model-cache\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.995986 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.995871    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-dshm\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.995986 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.995894    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-tls-certs\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.995986 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.995934    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-tmp-dir\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.996179 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.996029    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk8lv\" (UniqueName: \"kubernetes.io/projected/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-kube-api-access-lk8lv\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.996216 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.996196    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-home\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.996256 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.996234    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-model-cache\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.996462 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.996442    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-tmp-dir\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.998396 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.998371    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-dshm\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:58.998674 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:58.998659    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-tls-certs\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:59.007779 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:59.007758    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk8lv\" (UniqueName: \"kubernetes.io/projected/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-kube-api-access-lk8lv\") pod \"auth-test-model-a75117-kserve-946674bdc-l2466\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") " pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:59.170122 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:59.170033    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:29:59.453733 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:59.453703    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"]
Sep 18 16:29:59.455794 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:29:59.455760    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa5bd72f_b3fc_40f1_b9ce_eeaea9b802c8.slice/crio-98e5b01cf37805c5e606c0276da076308003988e25f69232aff11a644293a139 WatchSource:0}: Error finding container 98e5b01cf37805c5e606c0276da076308003988e25f69232aff11a644293a139: Status 404 returned error can't find the container with id 98e5b01cf37805c5e606c0276da076308003988e25f69232aff11a644293a139
Sep 18 16:29:59.551743 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:59.551709    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466" event={"ID":"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8","Type":"ContainerStarted","Data":"6fcb776b48605aa1182226f1b4f4e8607567176639396824dfe6e3110388249a"}
Sep 18 16:29:59.551870 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:59.551756    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466" event={"ID":"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8","Type":"ContainerStarted","Data":"98e5b01cf37805c5e606c0276da076308003988e25f69232aff11a644293a139"}
Sep 18 16:29:59.578714 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:29:59.578663    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466" podStartSLOduration=1.5786481669999999 podStartE2EDuration="1.578648167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:29:59.576361019 +0000 UTC m=+1494.523859793" watchObservedRunningTime="2026-09-18 16:29:59.578648167 +0000 UTC m=+1494.526146937"
Sep 18 16:30:00.200724 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:00.200688    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1eec1d-29829150-g2qjw"]
Sep 18 16:30:00.204736 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:00.204715    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1eec1d-29829150-g2qjw"
Sep 18 16:30:00.212793 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:00.212768    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:30:00.216286 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:00.216265    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1eec1d-29829150-g2qjw"
Sep 18 16:30:00.227206 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:00.227116    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1eec1d-29829150-g2qjw"]
Sep 18 16:30:00.238171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:00.238145    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829150-4rb4v"]
Sep 18 16:30:00.242654 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:00.242633    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829150-4rb4v"
Sep 18 16:30:00.256825 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:00.256801    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829150-4rb4v"
Sep 18 16:30:00.258334 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:00.258309    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829150-4rb4v"]
Sep 18 16:30:00.504127 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:00.504104    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1eec1d-29829150-g2qjw"]
Sep 18 16:30:00.506229 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:30:00.506202    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf246fb96_3d1d_4003_ba89_5441d87a7094.slice/crio-1d85ba76f475c473201cc9c6eff8ef78bf34e09f62ddb9b2a8e4ad3cf831f55b WatchSource:0}: Error finding container 1d85ba76f475c473201cc9c6eff8ef78bf34e09f62ddb9b2a8e4ad3cf831f55b: Status 404 returned error can't find the container with id 1d85ba76f475c473201cc9c6eff8ef78bf34e09f62ddb9b2a8e4ad3cf831f55b
Sep 18 16:30:00.558483 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:00.558457    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1eec1d-29829150-g2qjw" event={"ID":"f246fb96-3d1d-4003-ba89-5441d87a7094","Type":"ContainerStarted","Data":"1d85ba76f475c473201cc9c6eff8ef78bf34e09f62ddb9b2a8e4ad3cf831f55b"}
Sep 18 16:30:00.564176 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:00.564151    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829150-4rb4v"]
Sep 18 16:30:00.567255 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:30:00.567234    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba15230_a6ce_4138_818f_968f07086f41.slice/crio-9f22afc148ef4abedff03872698b2e62acea1f6c3302b6491272dc16804317f0 WatchSource:0}: Error finding container 9f22afc148ef4abedff03872698b2e62acea1f6c3302b6491272dc16804317f0: Status 404 returned error can't find the container with id 9f22afc148ef4abedff03872698b2e62acea1f6c3302b6491272dc16804317f0
Sep 18 16:30:01.565063 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:01.565028    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829150-4rb4v" event={"ID":"4ba15230-a6ce-4138-818f-968f07086f41","Type":"ContainerStarted","Data":"9f22afc148ef4abedff03872698b2e62acea1f6c3302b6491272dc16804317f0"}
Sep 18 16:30:03.580275 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:03.580234    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829150-4rb4v" event={"ID":"4ba15230-a6ce-4138-818f-968f07086f41","Type":"ContainerStarted","Data":"0cf0cb62076286fb7b2dbace4025ecbf9e6bc5504531dc9cb1a333cb54bc148e"}
Sep 18 16:30:03.581741 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:03.581717    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1eec1d-29829150-g2qjw" event={"ID":"f246fb96-3d1d-4003-ba89-5441d87a7094","Type":"ContainerStarted","Data":"90093d39e56536b53b081e98e0c3416a1e3f1d7bfb792a45369b78455ed237e5"}
Sep 18 16:30:03.601139 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:03.601056    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829150-4rb4v" podStartSLOduration=1.092168772 podStartE2EDuration="3.601037794s" totalImagesPullingTime="2.508869022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:30:03.600504329 +0000 UTC m=+1498.548003103" watchObservedRunningTime="2026-09-18 16:30:03.601037794 +0000 UTC m=+1498.548536568"
Sep 18 16:30:03.618348 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:03.618307    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1eec1d-29829150-g2qjw" podStartSLOduration=1.044173608 podStartE2EDuration="3.618296034s" totalImagesPullingTime="2.574122426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:30:03.616843583 +0000 UTC m=+1498.564342356" watchObservedRunningTime="2026-09-18 16:30:03.618296034 +0000 UTC m=+1498.565794808"
Sep 18 16:30:04.589179 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:04.589137    2554 generic.go:356] "Generic (PLEG): container finished" podID="4ba15230-a6ce-4138-818f-968f07086f41" containerID="0cf0cb62076286fb7b2dbace4025ecbf9e6bc5504531dc9cb1a333cb54bc148e" exitCode=0
Sep 18 16:30:04.589670 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:04.589217    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829150-4rb4v" event={"ID":"4ba15230-a6ce-4138-818f-968f07086f41","Type":"ContainerDied","Data":"0cf0cb62076286fb7b2dbace4025ecbf9e6bc5504531dc9cb1a333cb54bc148e"}
Sep 18 16:30:04.590750 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:04.590727    2554 generic.go:356] "Generic (PLEG): container finished" podID="f246fb96-3d1d-4003-ba89-5441d87a7094" containerID="90093d39e56536b53b081e98e0c3416a1e3f1d7bfb792a45369b78455ed237e5" exitCode=0
Sep 18 16:30:04.590854 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:04.590766    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1eec1d-29829150-g2qjw" event={"ID":"f246fb96-3d1d-4003-ba89-5441d87a7094","Type":"ContainerDied","Data":"90093d39e56536b53b081e98e0c3416a1e3f1d7bfb792a45369b78455ed237e5"}
Sep 18 16:30:05.739825 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:05.739798    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1eec1d-29829150-g2qjw"
Sep 18 16:30:05.760890 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:05.760869    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829150-4rb4v"
Sep 18 16:30:06.602385 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:06.602356    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829150-4rb4v"
Sep 18 16:30:06.602385 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:06.602367    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829150-4rb4v" event={"ID":"4ba15230-a6ce-4138-818f-968f07086f41","Type":"ContainerDied","Data":"9f22afc148ef4abedff03872698b2e62acea1f6c3302b6491272dc16804317f0"}
Sep 18 16:30:06.602651 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:06.602429    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f22afc148ef4abedff03872698b2e62acea1f6c3302b6491272dc16804317f0"
Sep 18 16:30:06.603873 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:06.603855    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1eec1d-29829150-g2qjw" event={"ID":"f246fb96-3d1d-4003-ba89-5441d87a7094","Type":"ContainerDied","Data":"1d85ba76f475c473201cc9c6eff8ef78bf34e09f62ddb9b2a8e4ad3cf831f55b"}
Sep 18 16:30:06.603950 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:06.603887    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d85ba76f475c473201cc9c6eff8ef78bf34e09f62ddb9b2a8e4ad3cf831f55b"
Sep 18 16:30:06.603950 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:06.603889    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1eec1d-29829150-g2qjw"
Sep 18 16:30:09.171029 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:09.170987    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:30:09.171405 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:09.171054    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:30:09.183864 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:09.183839    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:30:09.635043 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:09.635017    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:30:30.875783 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:30.875691    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"]
Sep 18 16:30:30.876226 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:30.875948    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p" podUID="1a0c52af-c871-41d1-b1be-d7e6d0359421" containerName="istio-proxy" containerID="cri-o://5d91c72a962ceecee30ea790e819a6ff52fff2bb65913c258e16770d1bfd90ce" gracePeriod=30
Sep 18 16:30:31.189275 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:31.189236    2554 patch_prober.go:28] interesting pod/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 16:30:31.189456 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:31.189299    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p" podUID="1a0c52af-c871-41d1-b1be-d7e6d0359421" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:30:36.124874 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.124847    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:30:36.257774 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.257739    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-envoy\") pod \"1a0c52af-c871-41d1-b1be-d7e6d0359421\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") "
Sep 18 16:30:36.257774 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.257783    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-credential-socket\") pod \"1a0c52af-c871-41d1-b1be-d7e6d0359421\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") "
Sep 18 16:30:36.258057 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.257806    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqvcc\" (UniqueName: \"kubernetes.io/projected/1a0c52af-c871-41d1-b1be-d7e6d0359421-kube-api-access-wqvcc\") pod \"1a0c52af-c871-41d1-b1be-d7e6d0359421\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") "
Sep 18 16:30:36.258057 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.257843    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-podinfo\") pod \"1a0c52af-c871-41d1-b1be-d7e6d0359421\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") "
Sep 18 16:30:36.258057 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.257874    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0c52af-c871-41d1-b1be-d7e6d0359421-istiod-ca-cert\") pod \"1a0c52af-c871-41d1-b1be-d7e6d0359421\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") "
Sep 18 16:30:36.258057 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.257935    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-data\") pod \"1a0c52af-c871-41d1-b1be-d7e6d0359421\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") "
Sep 18 16:30:36.258057 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.257981    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-workload-certs\") pod \"1a0c52af-c871-41d1-b1be-d7e6d0359421\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") "
Sep 18 16:30:36.258057 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.258048    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-workload-socket\") pod \"1a0c52af-c871-41d1-b1be-d7e6d0359421\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") "
Sep 18 16:30:36.258386 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.258114    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-token\") pod \"1a0c52af-c871-41d1-b1be-d7e6d0359421\" (UID: \"1a0c52af-c871-41d1-b1be-d7e6d0359421\") "
Sep 18 16:30:36.258386 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.258114    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a0c52af-c871-41d1-b1be-d7e6d0359421" (UID: "1a0c52af-c871-41d1-b1be-d7e6d0359421"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:30:36.258386 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.258346    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a0c52af-c871-41d1-b1be-d7e6d0359421" (UID: "1a0c52af-c871-41d1-b1be-d7e6d0359421"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:30:36.258865 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.258645    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a0c52af-c871-41d1-b1be-d7e6d0359421" (UID: "1a0c52af-c871-41d1-b1be-d7e6d0359421"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:30:36.258865 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.258730    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-data\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:30:36.258865 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.258757    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-credential-socket\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:30:36.258865 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.258811    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0c52af-c871-41d1-b1be-d7e6d0359421-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a0c52af-c871-41d1-b1be-d7e6d0359421" (UID: "1a0c52af-c871-41d1-b1be-d7e6d0359421"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:30:36.259262 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.259011    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a0c52af-c871-41d1-b1be-d7e6d0359421" (UID: "1a0c52af-c871-41d1-b1be-d7e6d0359421"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:30:36.260437 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.260405    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a0c52af-c871-41d1-b1be-d7e6d0359421" (UID: "1a0c52af-c871-41d1-b1be-d7e6d0359421"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:30:36.260819 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.260785    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0c52af-c871-41d1-b1be-d7e6d0359421-kube-api-access-wqvcc" (OuterVolumeSpecName: "kube-api-access-wqvcc") pod "1a0c52af-c871-41d1-b1be-d7e6d0359421" (UID: "1a0c52af-c871-41d1-b1be-d7e6d0359421"). InnerVolumeSpecName "kube-api-access-wqvcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:30:36.260974 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.260953    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a0c52af-c871-41d1-b1be-d7e6d0359421" (UID: "1a0c52af-c871-41d1-b1be-d7e6d0359421"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:30:36.261103 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.261086    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a0c52af-c871-41d1-b1be-d7e6d0359421" (UID: "1a0c52af-c871-41d1-b1be-d7e6d0359421"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:30:36.359748 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.359712    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0c52af-c871-41d1-b1be-d7e6d0359421-istiod-ca-cert\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:30:36.359748 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.359742    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-workload-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:30:36.359748 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.359750    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-workload-socket\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:30:36.359981 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.359759    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-token\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:30:36.359981 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.359768    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-envoy\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:30:36.359981 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.359776    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqvcc\" (UniqueName: \"kubernetes.io/projected/1a0c52af-c871-41d1-b1be-d7e6d0359421-kube-api-access-wqvcc\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:30:36.359981 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.359785    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0c52af-c871-41d1-b1be-d7e6d0359421-istio-podinfo\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:30:36.769121 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.769081    2554 generic.go:356] "Generic (PLEG): container finished" podID="1a0c52af-c871-41d1-b1be-d7e6d0359421" containerID="5d91c72a962ceecee30ea790e819a6ff52fff2bb65913c258e16770d1bfd90ce" exitCode=0
Sep 18 16:30:36.769295 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.769134    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"
Sep 18 16:30:36.769295 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.769165    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p" event={"ID":"1a0c52af-c871-41d1-b1be-d7e6d0359421","Type":"ContainerDied","Data":"5d91c72a962ceecee30ea790e819a6ff52fff2bb65913c258e16770d1bfd90ce"}
Sep 18 16:30:36.769295 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.769220    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p" event={"ID":"1a0c52af-c871-41d1-b1be-d7e6d0359421","Type":"ContainerDied","Data":"673de939ab47836d7395f35b12c9740152fcf048ee43500a3b00a1e09b4d38af"}
Sep 18 16:30:36.769295 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.769243    2554 scope.go:117] "RemoveContainer" containerID="5d91c72a962ceecee30ea790e819a6ff52fff2bb65913c258e16770d1bfd90ce"
Sep 18 16:30:36.778842 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.778823    2554 scope.go:117] "RemoveContainer" containerID="78532b4f0a1d39a6cb549df9a4bbf1f994ce14d2056016a9ed9670e8136ec842"
Sep 18 16:30:36.790837 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.790818    2554 scope.go:117] "RemoveContainer" containerID="5d91c72a962ceecee30ea790e819a6ff52fff2bb65913c258e16770d1bfd90ce"
Sep 18 16:30:36.791135 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:30:36.791112    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d91c72a962ceecee30ea790e819a6ff52fff2bb65913c258e16770d1bfd90ce\": container with ID starting with 5d91c72a962ceecee30ea790e819a6ff52fff2bb65913c258e16770d1bfd90ce not found: ID does not exist" containerID="5d91c72a962ceecee30ea790e819a6ff52fff2bb65913c258e16770d1bfd90ce"
Sep 18 16:30:36.791224 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.791143    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d91c72a962ceecee30ea790e819a6ff52fff2bb65913c258e16770d1bfd90ce"} err="failed to get container status \"5d91c72a962ceecee30ea790e819a6ff52fff2bb65913c258e16770d1bfd90ce\": rpc error: code = NotFound desc = could not find container \"5d91c72a962ceecee30ea790e819a6ff52fff2bb65913c258e16770d1bfd90ce\": container with ID starting with 5d91c72a962ceecee30ea790e819a6ff52fff2bb65913c258e16770d1bfd90ce not found: ID does not exist"
Sep 18 16:30:36.791224 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.791160    2554 scope.go:117] "RemoveContainer" containerID="78532b4f0a1d39a6cb549df9a4bbf1f994ce14d2056016a9ed9670e8136ec842"
Sep 18 16:30:36.791457 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:30:36.791433    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78532b4f0a1d39a6cb549df9a4bbf1f994ce14d2056016a9ed9670e8136ec842\": container with ID starting with 78532b4f0a1d39a6cb549df9a4bbf1f994ce14d2056016a9ed9670e8136ec842 not found: ID does not exist" containerID="78532b4f0a1d39a6cb549df9a4bbf1f994ce14d2056016a9ed9670e8136ec842"
Sep 18 16:30:36.791499 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.791467    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78532b4f0a1d39a6cb549df9a4bbf1f994ce14d2056016a9ed9670e8136ec842"} err="failed to get container status \"78532b4f0a1d39a6cb549df9a4bbf1f994ce14d2056016a9ed9670e8136ec842\": rpc error: code = NotFound desc = could not find container \"78532b4f0a1d39a6cb549df9a4bbf1f994ce14d2056016a9ed9670e8136ec842\": container with ID starting with 78532b4f0a1d39a6cb549df9a4bbf1f994ce14d2056016a9ed9670e8136ec842 not found: ID does not exist"
Sep 18 16:30:36.798653 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.798631    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"]
Sep 18 16:30:36.804819 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:36.804796    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-157a333a-openshift-default-855fdd4668-2gq2p"]
Sep 18 16:30:37.277985 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:30:37.277952    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a0c52af-c871-41d1-b1be-d7e6d0359421" path="/var/lib/kubelet/pods/1a0c52af-c871-41d1-b1be-d7e6d0359421/volumes"
Sep 18 16:31:15.064498 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.064459    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"]
Sep 18 16:31:15.064980 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.064965    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba15230-a6ce-4138-818f-968f07086f41" containerName="cleanup"
Sep 18 16:31:15.065024 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.064982    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba15230-a6ce-4138-818f-968f07086f41" containerName="cleanup"
Sep 18 16:31:15.065024 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.064994    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f246fb96-3d1d-4003-ba89-5441d87a7094" containerName="cleanup"
Sep 18 16:31:15.065024 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.064999    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f246fb96-3d1d-4003-ba89-5441d87a7094" containerName="cleanup"
Sep 18 16:31:15.065024 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.065013    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a0c52af-c871-41d1-b1be-d7e6d0359421" containerName="istio-proxy"
Sep 18 16:31:15.065024 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.065018    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a0c52af-c871-41d1-b1be-d7e6d0359421" containerName="istio-proxy"
Sep 18 16:31:15.065181 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.065035    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a0c52af-c871-41d1-b1be-d7e6d0359421" containerName="istio-proxy"
Sep 18 16:31:15.065181 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.065044    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a0c52af-c871-41d1-b1be-d7e6d0359421" containerName="istio-proxy"
Sep 18 16:31:15.065181 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.065124    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a0c52af-c871-41d1-b1be-d7e6d0359421" containerName="istio-proxy"
Sep 18 16:31:15.065181 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.065138    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f246fb96-3d1d-4003-ba89-5441d87a7094" containerName="cleanup"
Sep 18 16:31:15.065181 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.065145    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba15230-a6ce-4138-818f-968f07086f41" containerName="cleanup"
Sep 18 16:31:15.069767 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.069744    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.073586 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.073562    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-11725e\"/\"openshift-service-ca.crt\""
Sep 18 16:31:15.074491 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.074472    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-11725e\"/\"kube-root-ca.crt\""
Sep 18 16:31:15.074842 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.074823    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-11725e\"/\"default-dockercfg-8bsfj\""
Sep 18 16:31:15.074923 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.074857    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-11725e\"/\"auth-test-model-11725e-kserve-self-signed-certs\""
Sep 18 16:31:15.082293 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.082266    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"]
Sep 18 16:31:15.219941 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.219906    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f6471d-0cdc-4796-92f9-33e223906222-tls-certs\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.220133 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.220031    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-home\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.220133 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.220062    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9xwd\" (UniqueName: \"kubernetes.io/projected/56f6471d-0cdc-4796-92f9-33e223906222-kube-api-access-w9xwd\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.220133 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.220102    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-tmp-dir\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.220249 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.220129    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-dshm\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.220249 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.220179    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-model-cache\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.321829 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.321729    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-home\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.321829 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.321774    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9xwd\" (UniqueName: \"kubernetes.io/projected/56f6471d-0cdc-4796-92f9-33e223906222-kube-api-access-w9xwd\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.321829 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.321804    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-tmp-dir\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.321829 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.321821    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-dshm\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.322139 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.321863    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-model-cache\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.322139 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.321894    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f6471d-0cdc-4796-92f9-33e223906222-tls-certs\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.322208 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.322167    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-home\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.322246 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.322216    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-tmp-dir\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.322281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.322256    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-model-cache\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.324367 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.324342    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-dshm\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.324683 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.324665    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f6471d-0cdc-4796-92f9-33e223906222-tls-certs\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.332955 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.332929    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9xwd\" (UniqueName: \"kubernetes.io/projected/56f6471d-0cdc-4796-92f9-33e223906222-kube-api-access-w9xwd\") pod \"auth-test-model-11725e-kserve-67c56d9947-5zg7r\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") " pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.379867 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.379805    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:15.657743 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.657713    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"]
Sep 18 16:31:15.659257 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:31:15.659225    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56f6471d_0cdc_4796_92f9_33e223906222.slice/crio-f4df13efb4971228934bb331034049ae16d0f160ccd91acdd7dc9c527b04a7bc WatchSource:0}: Error finding container f4df13efb4971228934bb331034049ae16d0f160ccd91acdd7dc9c527b04a7bc: Status 404 returned error can't find the container with id f4df13efb4971228934bb331034049ae16d0f160ccd91acdd7dc9c527b04a7bc
Sep 18 16:31:15.976210 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.976175    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r" event={"ID":"56f6471d-0cdc-4796-92f9-33e223906222","Type":"ContainerStarted","Data":"315a9bbd3f109d25683002c5d04a31ab491ae261742a6a97c74812bc896fcc9c"}
Sep 18 16:31:15.976210 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:15.976217    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r" event={"ID":"56f6471d-0cdc-4796-92f9-33e223906222","Type":"ContainerStarted","Data":"f4df13efb4971228934bb331034049ae16d0f160ccd91acdd7dc9c527b04a7bc"}
Sep 18 16:31:16.043294 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:16.043223    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r" podStartSLOduration=1.043205074 podStartE2EDuration="1.043205074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:31:16.041297777 +0000 UTC m=+1570.988796569" watchObservedRunningTime="2026-09-18 16:31:16.043205074 +0000 UTC m=+1570.990703848"
Sep 18 16:31:25.380452 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:25.380417    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:25.380844 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:25.380476    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:25.392672 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:25.392645    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:26.039704 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:26.039669    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:31:29.354706 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.354669    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"]
Sep 18 16:31:29.355191 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.355178    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a0c52af-c871-41d1-b1be-d7e6d0359421" containerName="istio-proxy"
Sep 18 16:31:29.358297 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.358280    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.365418 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.365385    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ac5968-openshift-default-dockercfg-27b5q\""
Sep 18 16:31:29.380148 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.380111    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"]
Sep 18 16:31:29.468090 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.468057    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-credential-socket\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.468090 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.468102    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lktbc\" (UniqueName: \"kubernetes.io/projected/05a57950-a7f7-424c-8b42-542567916e3b-kube-api-access-lktbc\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.468326 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.468186    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a57950-a7f7-424c-8b42-542567916e3b-istio-token\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.468326 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.468224    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-workload-socket\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.468326 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.468253    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-istio-data\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.468326 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.468308    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a57950-a7f7-424c-8b42-542567916e3b-istio-podinfo\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.468326 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.468323    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a57950-a7f7-424c-8b42-542567916e3b-istiod-ca-cert\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.468785 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.468355    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-workload-certs\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.468785 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.468468    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-istio-envoy\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.569296 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.569256    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-istio-envoy\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.569508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.569326    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-credential-socket\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.569508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.569344    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lktbc\" (UniqueName: \"kubernetes.io/projected/05a57950-a7f7-424c-8b42-542567916e3b-kube-api-access-lktbc\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.569508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.569372    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a57950-a7f7-424c-8b42-542567916e3b-istio-token\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.569508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.569427    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-workload-socket\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.569508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.569450    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-istio-data\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.569508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.569487    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a57950-a7f7-424c-8b42-542567916e3b-istio-podinfo\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.569858 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.569512    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a57950-a7f7-424c-8b42-542567916e3b-istiod-ca-cert\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.569858 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.569584    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-workload-certs\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.569974 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.569892    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-istio-data\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.570032 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.569966    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-workload-socket\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.570032 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.570005    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-credential-socket\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.570137 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.570092    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-workload-certs\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.570328 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.570310    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a57950-a7f7-424c-8b42-542567916e3b-istiod-ca-cert\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.571817 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.571796    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-istio-envoy\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.572157 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.572135    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a57950-a7f7-424c-8b42-542567916e3b-istio-podinfo\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.582674 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.582649    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a57950-a7f7-424c-8b42-542567916e3b-istio-token\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.583635 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.583612    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lktbc\" (UniqueName: \"kubernetes.io/projected/05a57950-a7f7-424c-8b42-542567916e3b-kube-api-access-lktbc\") pod \"e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") " pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.669560 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.669427    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:29.959116 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.959091    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"]
Sep 18 16:31:29.959920 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:31:29.959896    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a57950_a7f7_424c_8b42_542567916e3b.slice/crio-dfe81a2f5a700a92d46252f3dff735e378cbd53ca68170bc23a670e2e2d289d7 WatchSource:0}: Error finding container dfe81a2f5a700a92d46252f3dff735e378cbd53ca68170bc23a670e2e2d289d7: Status 404 returned error can't find the container with id dfe81a2f5a700a92d46252f3dff735e378cbd53ca68170bc23a670e2e2d289d7
Sep 18 16:31:29.962243 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.962207    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:31:29.962313 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.962274    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:31:29.962313 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:29.962303    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:31:30.052104 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:30.052065    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx" event={"ID":"05a57950-a7f7-424c-8b42-542567916e3b","Type":"ContainerStarted","Data":"e6d21fe10ec4012b479c0381b484c0ccb1b518bfc8bd3693aff2f7a14576b639"}
Sep 18 16:31:30.052288 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:30.052131    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx" event={"ID":"05a57950-a7f7-424c-8b42-542567916e3b","Type":"ContainerStarted","Data":"dfe81a2f5a700a92d46252f3dff735e378cbd53ca68170bc23a670e2e2d289d7"}
Sep 18 16:31:30.099026 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:30.098956    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx" podStartSLOduration=1.098934531 podStartE2EDuration="1.098934531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:31:30.098398204 +0000 UTC m=+1585.045896977" watchObservedRunningTime="2026-09-18 16:31:30.098934531 +0000 UTC m=+1585.046433322"
Sep 18 16:31:30.670535 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:30.670501    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:31.675061 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:31.675029    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:32.062823 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:32.062728    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:32.063746 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:32.063716    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:31:42.205137 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.205106    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"]
Sep 18 16:31:42.209205 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.209181    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"
Sep 18 16:31:42.211558 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.211516    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"]
Sep 18 16:31:42.211983 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.211964    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ac5968-dockercfg-5sj9w\""
Sep 18 16:31:42.212208 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.212194    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ac5968\""
Sep 18 16:31:42.215072 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.215058    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"
Sep 18 16:31:42.236406 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.236384    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"]
Sep 18 16:31:42.246182 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.246157    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"]
Sep 18 16:31:42.305470 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.305430    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prgn9\" (UniqueName: \"kubernetes.io/projected/04710d80-2eff-4dcc-af26-51c99ef34331-kube-api-access-prgn9\") pod \"payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl\" (UID: \"04710d80-2eff-4dcc-af26-51c99ef34331\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"
Sep 18 16:31:42.305675 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.305555    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e82c812-598f-46d3-9c31-a52fb6a37b92-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j\" (UID: \"8e82c812-598f-46d3-9c31-a52fb6a37b92\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"
Sep 18 16:31:42.305675 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.305600    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlgl2\" (UniqueName: \"kubernetes.io/projected/8e82c812-598f-46d3-9c31-a52fb6a37b92-kube-api-access-qlgl2\") pod \"payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j\" (UID: \"8e82c812-598f-46d3-9c31-a52fb6a37b92\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"
Sep 18 16:31:42.305675 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.305657    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04710d80-2eff-4dcc-af26-51c99ef34331-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl\" (UID: \"04710d80-2eff-4dcc-af26-51c99ef34331\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"
Sep 18 16:31:42.406314 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.406280    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prgn9\" (UniqueName: \"kubernetes.io/projected/04710d80-2eff-4dcc-af26-51c99ef34331-kube-api-access-prgn9\") pod \"payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl\" (UID: \"04710d80-2eff-4dcc-af26-51c99ef34331\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"
Sep 18 16:31:42.406550 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.406411    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e82c812-598f-46d3-9c31-a52fb6a37b92-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j\" (UID: \"8e82c812-598f-46d3-9c31-a52fb6a37b92\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"
Sep 18 16:31:42.406550 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.406474    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlgl2\" (UniqueName: \"kubernetes.io/projected/8e82c812-598f-46d3-9c31-a52fb6a37b92-kube-api-access-qlgl2\") pod \"payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j\" (UID: \"8e82c812-598f-46d3-9c31-a52fb6a37b92\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"
Sep 18 16:31:42.406713 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.406685    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04710d80-2eff-4dcc-af26-51c99ef34331-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl\" (UID: \"04710d80-2eff-4dcc-af26-51c99ef34331\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"
Sep 18 16:31:42.407240 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.407221    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e82c812-598f-46d3-9c31-a52fb6a37b92-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j\" (UID: \"8e82c812-598f-46d3-9c31-a52fb6a37b92\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"
Sep 18 16:31:42.407411 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.407394    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04710d80-2eff-4dcc-af26-51c99ef34331-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl\" (UID: \"04710d80-2eff-4dcc-af26-51c99ef34331\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"
Sep 18 16:31:42.419850 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.419816    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlgl2\" (UniqueName: \"kubernetes.io/projected/8e82c812-598f-46d3-9c31-a52fb6a37b92-kube-api-access-qlgl2\") pod \"payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j\" (UID: \"8e82c812-598f-46d3-9c31-a52fb6a37b92\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"
Sep 18 16:31:42.420028 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.420006    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prgn9\" (UniqueName: \"kubernetes.io/projected/04710d80-2eff-4dcc-af26-51c99ef34331-kube-api-access-prgn9\") pod \"payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl\" (UID: \"04710d80-2eff-4dcc-af26-51c99ef34331\") " pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"
Sep 18 16:31:42.521588 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.521461    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"
Sep 18 16:31:42.526886 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.526852    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"
Sep 18 16:31:42.832098 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.832067    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"]
Sep 18 16:31:42.834128 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:42.834085    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"]
Sep 18 16:31:42.834616 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:31:42.834593    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04710d80_2eff_4dcc_af26_51c99ef34331.slice/crio-13d8e7ab792d1d0bc2e9200e1990f4be8d9a70a92bbe95f8f53c68be60d5666e WatchSource:0}: Error finding container 13d8e7ab792d1d0bc2e9200e1990f4be8d9a70a92bbe95f8f53c68be60d5666e: Status 404 returned error can't find the container with id 13d8e7ab792d1d0bc2e9200e1990f4be8d9a70a92bbe95f8f53c68be60d5666e
Sep 18 16:31:42.835484 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:31:42.835369    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e82c812_598f_46d3_9c31_a52fb6a37b92.slice/crio-f2585cdc8da9995cd1c9617e5540dc7abd57cf60707c332232d621045d5cf53e WatchSource:0}: Error finding container f2585cdc8da9995cd1c9617e5540dc7abd57cf60707c332232d621045d5cf53e: Status 404 returned error can't find the container with id f2585cdc8da9995cd1c9617e5540dc7abd57cf60707c332232d621045d5cf53e
Sep 18 16:31:43.124234 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:43.124138    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl" event={"ID":"04710d80-2eff-4dcc-af26-51c99ef34331","Type":"ContainerStarted","Data":"6797c5ba084bbf4df58f03b5b0e33c0239e77f9eaec64c629946c9c7f8e95de1"}
Sep 18 16:31:43.124234 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:43.124188    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl" event={"ID":"04710d80-2eff-4dcc-af26-51c99ef34331","Type":"ContainerStarted","Data":"13d8e7ab792d1d0bc2e9200e1990f4be8d9a70a92bbe95f8f53c68be60d5666e"}
Sep 18 16:31:43.124466 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:43.124300    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"
Sep 18 16:31:43.125923 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:43.125898    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j" event={"ID":"8e82c812-598f-46d3-9c31-a52fb6a37b92","Type":"ContainerStarted","Data":"513a451970ff79a11467058a1be6d2e454ae0e8668a8d64060fbe1e8f883f2cd"}
Sep 18 16:31:43.126047 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:43.125949    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"
Sep 18 16:31:43.126047 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:43.125973    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j" event={"ID":"8e82c812-598f-46d3-9c31-a52fb6a37b92","Type":"ContainerStarted","Data":"f2585cdc8da9995cd1c9617e5540dc7abd57cf60707c332232d621045d5cf53e"}
Sep 18 16:31:43.151682 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:43.151639    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl" podStartSLOduration=1.151626328 podStartE2EDuration="1.151626328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:31:43.150108837 +0000 UTC m=+1598.097607615" watchObservedRunningTime="2026-09-18 16:31:43.151626328 +0000 UTC m=+1598.099125102"
Sep 18 16:31:43.186843 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:43.186801    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j" podStartSLOduration=1.18678952 podStartE2EDuration="1.18678952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:31:43.185030431 +0000 UTC m=+1598.132529205" watchObservedRunningTime="2026-09-18 16:31:43.18678952 +0000 UTC m=+1598.134288328"
Sep 18 16:31:54.133449 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:54.133372    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"
Sep 18 16:31:54.133449 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:31:54.133442    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"
Sep 18 16:32:15.129339 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:15.129297    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"]
Sep 18 16:32:15.133409 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:15.133385    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"
Sep 18 16:32:15.140361 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:15.140340    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9eae22\""
Sep 18 16:32:15.140691 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:15.140672    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9eae22-dockercfg-vmkc2\""
Sep 18 16:32:15.147592 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:15.147566    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"]
Sep 18 16:32:15.229988 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:15.229963    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35b8793-64a1-4d07-801e-f9aa31da7fc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst\" (UID: \"f35b8793-64a1-4d07-801e-f9aa31da7fc2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"
Sep 18 16:32:15.232036 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:15.230023    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx6jn\" (UniqueName: \"kubernetes.io/projected/f35b8793-64a1-4d07-801e-f9aa31da7fc2-kube-api-access-sx6jn\") pod \"payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst\" (UID: \"f35b8793-64a1-4d07-801e-f9aa31da7fc2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"
Sep 18 16:32:15.330658 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:15.330634    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx6jn\" (UniqueName: \"kubernetes.io/projected/f35b8793-64a1-4d07-801e-f9aa31da7fc2-kube-api-access-sx6jn\") pod \"payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst\" (UID: \"f35b8793-64a1-4d07-801e-f9aa31da7fc2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"
Sep 18 16:32:15.330775 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:15.330737    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35b8793-64a1-4d07-801e-f9aa31da7fc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst\" (UID: \"f35b8793-64a1-4d07-801e-f9aa31da7fc2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"
Sep 18 16:32:15.331349 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:15.331331    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35b8793-64a1-4d07-801e-f9aa31da7fc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst\" (UID: \"f35b8793-64a1-4d07-801e-f9aa31da7fc2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"
Sep 18 16:32:15.345307 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:15.345283    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx6jn\" (UniqueName: \"kubernetes.io/projected/f35b8793-64a1-4d07-801e-f9aa31da7fc2-kube-api-access-sx6jn\") pod \"payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst\" (UID: \"f35b8793-64a1-4d07-801e-f9aa31da7fc2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"
Sep 18 16:32:15.446961 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:15.446943    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"
Sep 18 16:32:15.728469 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:15.728401    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"]
Sep 18 16:32:15.731934 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:32:15.731906    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35b8793_64a1_4d07_801e_f9aa31da7fc2.slice/crio-bedf4f2019a18f298cf6e8cacc7f335c1a7e97af70623a261ccfb2dc1946c940 WatchSource:0}: Error finding container bedf4f2019a18f298cf6e8cacc7f335c1a7e97af70623a261ccfb2dc1946c940: Status 404 returned error can't find the container with id bedf4f2019a18f298cf6e8cacc7f335c1a7e97af70623a261ccfb2dc1946c940
Sep 18 16:32:16.311497 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:16.311465    2554 generic.go:356] "Generic (PLEG): container finished" podID="05a57950-a7f7-424c-8b42-542567916e3b" containerID="e6d21fe10ec4012b479c0381b484c0ccb1b518bfc8bd3693aff2f7a14576b639" exitCode=0
Sep 18 16:32:16.312001 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:16.311558    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx" event={"ID":"05a57950-a7f7-424c-8b42-542567916e3b","Type":"ContainerDied","Data":"e6d21fe10ec4012b479c0381b484c0ccb1b518bfc8bd3693aff2f7a14576b639"}
Sep 18 16:32:16.312065 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:16.312014    2554 scope.go:117] "RemoveContainer" containerID="e6d21fe10ec4012b479c0381b484c0ccb1b518bfc8bd3693aff2f7a14576b639"
Sep 18 16:32:16.313340 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:16.313314    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst" event={"ID":"f35b8793-64a1-4d07-801e-f9aa31da7fc2","Type":"ContainerStarted","Data":"9a284650a7be00d49a723a5c8a5416dc56d06f3f9dd70da3c5f519f9f92131e0"}
Sep 18 16:32:16.313452 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:16.313364    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst" event={"ID":"f35b8793-64a1-4d07-801e-f9aa31da7fc2","Type":"ContainerStarted","Data":"bedf4f2019a18f298cf6e8cacc7f335c1a7e97af70623a261ccfb2dc1946c940"}
Sep 18 16:32:16.313452 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:16.313405    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"
Sep 18 16:32:16.313452 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:16.313406    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:32:16.313594 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:16.313478    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:32:16.313594 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:16.313517    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 18 16:32:16.377049 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:16.377002    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst" podStartSLOduration=1.376984897 podStartE2EDuration="1.376984897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:32:16.376699502 +0000 UTC m=+1631.324198296" watchObservedRunningTime="2026-09-18 16:32:16.376984897 +0000 UTC m=+1631.324483694"
Sep 18 16:32:17.063066 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:17.063036    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:32:17.319876 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:17.319779    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx" event={"ID":"05a57950-a7f7-424c-8b42-542567916e3b","Type":"ContainerStarted","Data":"887f49b4d9fa00ec044f267df5f3f01a6f202ff6ac50db6b69b45847b769de3f"}
Sep 18 16:32:17.669768 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:17.669668    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:32:17.671294 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:17.671265    2554 patch_prober.go:28] interesting pod/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 18 16:32:17.671565 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:17.671508    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx" podUID="05a57950-a7f7-424c-8b42-542567916e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 18 16:32:18.670379 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:18.670346    2554 patch_prober.go:28] interesting pod/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 18 16:32:18.670822 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:18.670400    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx" podUID="05a57950-a7f7-424c-8b42-542567916e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 18 16:32:19.670636 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:19.670600    2554 patch_prober.go:28] interesting pod/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 18 16:32:19.671051 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:19.670669    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx" podUID="05a57950-a7f7-424c-8b42-542567916e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 18 16:32:20.670290 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:20.670253    2554 patch_prober.go:28] interesting pod/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 18 16:32:20.670475 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:20.670324    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx" podUID="05a57950-a7f7-424c-8b42-542567916e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 18 16:32:21.670792 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:21.670753    2554 patch_prober.go:28] interesting pod/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 18 16:32:21.671229 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:21.670812    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx" podUID="05a57950-a7f7-424c-8b42-542567916e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 18 16:32:22.670744 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:22.670709    2554 patch_prober.go:28] interesting pod/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 18 16:32:22.670925 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:22.670770    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx" podUID="05a57950-a7f7-424c-8b42-542567916e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 18 16:32:23.675479 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:23.675446    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:32:23.675913 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:23.675848    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:32:23.676581 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:23.676562    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:32:27.321969 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:27.321939    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"
Sep 18 16:32:45.543337 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:45.543287    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"]
Sep 18 16:32:45.543852 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:45.543588    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl" podUID="04710d80-2eff-4dcc-af26-51c99ef34331" containerName="payload-processing" containerID="cri-o://6797c5ba084bbf4df58f03b5b0e33c0239e77f9eaec64c629946c9c7f8e95de1" gracePeriod=30
Sep 18 16:32:45.546212 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:45.546185    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"]
Sep 18 16:32:45.546601 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:45.546569    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j" podUID="8e82c812-598f-46d3-9c31-a52fb6a37b92" containerName="payload-pre-processing" containerID="cri-o://513a451970ff79a11467058a1be6d2e454ae0e8668a8d64060fbe1e8f883f2cd" gracePeriod=30
Sep 18 16:32:45.822440 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:45.822397    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"
Sep 18 16:32:45.835804 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:45.835776    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"
Sep 18 16:32:45.939871 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:45.939834    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlgl2\" (UniqueName: \"kubernetes.io/projected/8e82c812-598f-46d3-9c31-a52fb6a37b92-kube-api-access-qlgl2\") pod \"8e82c812-598f-46d3-9c31-a52fb6a37b92\" (UID: \"8e82c812-598f-46d3-9c31-a52fb6a37b92\") "
Sep 18 16:32:45.940057 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:45.939916    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e82c812-598f-46d3-9c31-a52fb6a37b92-plugins-config-volume\") pod \"8e82c812-598f-46d3-9c31-a52fb6a37b92\" (UID: \"8e82c812-598f-46d3-9c31-a52fb6a37b92\") "
Sep 18 16:32:45.940057 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:45.939955    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04710d80-2eff-4dcc-af26-51c99ef34331-plugins-config-volume\") pod \"04710d80-2eff-4dcc-af26-51c99ef34331\" (UID: \"04710d80-2eff-4dcc-af26-51c99ef34331\") "
Sep 18 16:32:45.940184 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:45.940074    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prgn9\" (UniqueName: \"kubernetes.io/projected/04710d80-2eff-4dcc-af26-51c99ef34331-kube-api-access-prgn9\") pod \"04710d80-2eff-4dcc-af26-51c99ef34331\" (UID: \"04710d80-2eff-4dcc-af26-51c99ef34331\") "
Sep 18 16:32:45.940354 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:45.940325    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e82c812-598f-46d3-9c31-a52fb6a37b92-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e82c812-598f-46d3-9c31-a52fb6a37b92" (UID: "8e82c812-598f-46d3-9c31-a52fb6a37b92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:32:45.940486 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:45.940463    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e82c812-598f-46d3-9c31-a52fb6a37b92-plugins-config-volume\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:32:45.940744 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:45.940637    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04710d80-2eff-4dcc-af26-51c99ef34331-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "04710d80-2eff-4dcc-af26-51c99ef34331" (UID: "04710d80-2eff-4dcc-af26-51c99ef34331"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:32:45.943192 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:45.943163    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04710d80-2eff-4dcc-af26-51c99ef34331-kube-api-access-prgn9" (OuterVolumeSpecName: "kube-api-access-prgn9") pod "04710d80-2eff-4dcc-af26-51c99ef34331" (UID: "04710d80-2eff-4dcc-af26-51c99ef34331"). InnerVolumeSpecName "kube-api-access-prgn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:32:45.943391 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:45.943368    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e82c812-598f-46d3-9c31-a52fb6a37b92-kube-api-access-qlgl2" (OuterVolumeSpecName: "kube-api-access-qlgl2") pod "8e82c812-598f-46d3-9c31-a52fb6a37b92" (UID: "8e82c812-598f-46d3-9c31-a52fb6a37b92"). InnerVolumeSpecName "kube-api-access-qlgl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:32:46.041104 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.041068    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prgn9\" (UniqueName: \"kubernetes.io/projected/04710d80-2eff-4dcc-af26-51c99ef34331-kube-api-access-prgn9\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:32:46.041104 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.041099    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlgl2\" (UniqueName: \"kubernetes.io/projected/8e82c812-598f-46d3-9c31-a52fb6a37b92-kube-api-access-qlgl2\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:32:46.041104 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.041108    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04710d80-2eff-4dcc-af26-51c99ef34331-plugins-config-volume\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:32:46.489259 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.489224    2554 generic.go:356] "Generic (PLEG): container finished" podID="8e82c812-598f-46d3-9c31-a52fb6a37b92" containerID="513a451970ff79a11467058a1be6d2e454ae0e8668a8d64060fbe1e8f883f2cd" exitCode=0
Sep 18 16:32:46.489426 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.489266    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"
Sep 18 16:32:46.489426 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.489309    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j" event={"ID":"8e82c812-598f-46d3-9c31-a52fb6a37b92","Type":"ContainerDied","Data":"513a451970ff79a11467058a1be6d2e454ae0e8668a8d64060fbe1e8f883f2cd"}
Sep 18 16:32:46.489426 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.489361    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j" event={"ID":"8e82c812-598f-46d3-9c31-a52fb6a37b92","Type":"ContainerDied","Data":"f2585cdc8da9995cd1c9617e5540dc7abd57cf60707c332232d621045d5cf53e"}
Sep 18 16:32:46.489426 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.489395    2554 scope.go:117] "RemoveContainer" containerID="513a451970ff79a11467058a1be6d2e454ae0e8668a8d64060fbe1e8f883f2cd"
Sep 18 16:32:46.490993 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.490967    2554 generic.go:356] "Generic (PLEG): container finished" podID="04710d80-2eff-4dcc-af26-51c99ef34331" containerID="6797c5ba084bbf4df58f03b5b0e33c0239e77f9eaec64c629946c9c7f8e95de1" exitCode=0
Sep 18 16:32:46.491125 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.491020    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"
Sep 18 16:32:46.491125 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.491048    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl" event={"ID":"04710d80-2eff-4dcc-af26-51c99ef34331","Type":"ContainerDied","Data":"6797c5ba084bbf4df58f03b5b0e33c0239e77f9eaec64c629946c9c7f8e95de1"}
Sep 18 16:32:46.491125 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.491100    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl" event={"ID":"04710d80-2eff-4dcc-af26-51c99ef34331","Type":"ContainerDied","Data":"13d8e7ab792d1d0bc2e9200e1990f4be8d9a70a92bbe95f8f53c68be60d5666e"}
Sep 18 16:32:46.499201 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.499180    2554 scope.go:117] "RemoveContainer" containerID="513a451970ff79a11467058a1be6d2e454ae0e8668a8d64060fbe1e8f883f2cd"
Sep 18 16:32:46.499539 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:32:46.499498    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"513a451970ff79a11467058a1be6d2e454ae0e8668a8d64060fbe1e8f883f2cd\": container with ID starting with 513a451970ff79a11467058a1be6d2e454ae0e8668a8d64060fbe1e8f883f2cd not found: ID does not exist" containerID="513a451970ff79a11467058a1be6d2e454ae0e8668a8d64060fbe1e8f883f2cd"
Sep 18 16:32:46.499692 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.499661    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"513a451970ff79a11467058a1be6d2e454ae0e8668a8d64060fbe1e8f883f2cd"} err="failed to get container status \"513a451970ff79a11467058a1be6d2e454ae0e8668a8d64060fbe1e8f883f2cd\": rpc error: code = NotFound desc = could not find container \"513a451970ff79a11467058a1be6d2e454ae0e8668a8d64060fbe1e8f883f2cd\": container with ID starting with 513a451970ff79a11467058a1be6d2e454ae0e8668a8d64060fbe1e8f883f2cd not found: ID does not exist"
Sep 18 16:32:46.499692 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.499694    2554 scope.go:117] "RemoveContainer" containerID="6797c5ba084bbf4df58f03b5b0e33c0239e77f9eaec64c629946c9c7f8e95de1"
Sep 18 16:32:46.507843 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.507826    2554 scope.go:117] "RemoveContainer" containerID="6797c5ba084bbf4df58f03b5b0e33c0239e77f9eaec64c629946c9c7f8e95de1"
Sep 18 16:32:46.508079 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:32:46.508060    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6797c5ba084bbf4df58f03b5b0e33c0239e77f9eaec64c629946c9c7f8e95de1\": container with ID starting with 6797c5ba084bbf4df58f03b5b0e33c0239e77f9eaec64c629946c9c7f8e95de1 not found: ID does not exist" containerID="6797c5ba084bbf4df58f03b5b0e33c0239e77f9eaec64c629946c9c7f8e95de1"
Sep 18 16:32:46.508168 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.508085    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6797c5ba084bbf4df58f03b5b0e33c0239e77f9eaec64c629946c9c7f8e95de1"} err="failed to get container status \"6797c5ba084bbf4df58f03b5b0e33c0239e77f9eaec64c629946c9c7f8e95de1\": rpc error: code = NotFound desc = could not find container \"6797c5ba084bbf4df58f03b5b0e33c0239e77f9eaec64c629946c9c7f8e95de1\": container with ID starting with 6797c5ba084bbf4df58f03b5b0e33c0239e77f9eaec64c629946c9c7f8e95de1 not found: ID does not exist"
Sep 18 16:32:46.524653 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.524627    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"]
Sep 18 16:32:46.526814 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.526792    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ac5968-fb68d4d5f-4qgfl"]
Sep 18 16:32:46.542351 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.542326    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"]
Sep 18 16:32:46.554900 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:46.554878    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ac5968-654df6f566-8579j"]
Sep 18 16:32:47.277827 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:47.277793    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04710d80-2eff-4dcc-af26-51c99ef34331" path="/var/lib/kubelet/pods/04710d80-2eff-4dcc-af26-51c99ef34331/volumes"
Sep 18 16:32:47.278177 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:47.278164    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e82c812-598f-46d3-9c31-a52fb6a37b92" path="/var/lib/kubelet/pods/8e82c812-598f-46d3-9c31-a52fb6a37b92/volumes"
Sep 18 16:32:54.743873 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:54.743842    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"]
Sep 18 16:32:54.744277 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:54.744039    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx" podUID="05a57950-a7f7-424c-8b42-542567916e3b" containerName="istio-proxy" containerID="cri-o://887f49b4d9fa00ec044f267df5f3f01a6f202ff6ac50db6b69b45847b769de3f" gracePeriod=30
Sep 18 16:32:59.996785 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:32:59.996759    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:33:00.084935 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.084857    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-credential-socket\") pod \"05a57950-a7f7-424c-8b42-542567916e3b\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") "
Sep 18 16:33:00.084935 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.084917    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-workload-certs\") pod \"05a57950-a7f7-424c-8b42-542567916e3b\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") "
Sep 18 16:33:00.085160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.084947    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a57950-a7f7-424c-8b42-542567916e3b-istio-podinfo\") pod \"05a57950-a7f7-424c-8b42-542567916e3b\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") "
Sep 18 16:33:00.085160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.084965    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a57950-a7f7-424c-8b42-542567916e3b-istiod-ca-cert\") pod \"05a57950-a7f7-424c-8b42-542567916e3b\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") "
Sep 18 16:33:00.085160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.084993    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lktbc\" (UniqueName: \"kubernetes.io/projected/05a57950-a7f7-424c-8b42-542567916e3b-kube-api-access-lktbc\") pod \"05a57950-a7f7-424c-8b42-542567916e3b\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") "
Sep 18 16:33:00.085160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.085020    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-workload-socket\") pod \"05a57950-a7f7-424c-8b42-542567916e3b\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") "
Sep 18 16:33:00.085160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.085043    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-istio-data\") pod \"05a57950-a7f7-424c-8b42-542567916e3b\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") "
Sep 18 16:33:00.085160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.085125    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a57950-a7f7-424c-8b42-542567916e3b-istio-token\") pod \"05a57950-a7f7-424c-8b42-542567916e3b\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") "
Sep 18 16:33:00.085160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.085153    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-istio-envoy\") pod \"05a57950-a7f7-424c-8b42-542567916e3b\" (UID: \"05a57950-a7f7-424c-8b42-542567916e3b\") "
Sep 18 16:33:00.085472 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.085164    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "05a57950-a7f7-424c-8b42-542567916e3b" (UID: "05a57950-a7f7-424c-8b42-542567916e3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:33:00.085472 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.085383    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "05a57950-a7f7-424c-8b42-542567916e3b" (UID: "05a57950-a7f7-424c-8b42-542567916e3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:33:00.085472 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.085437    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a57950-a7f7-424c-8b42-542567916e3b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "05a57950-a7f7-424c-8b42-542567916e3b" (UID: "05a57950-a7f7-424c-8b42-542567916e3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:33:00.085686 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.085563    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-credential-socket\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:00.085686 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.085582    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-workload-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:00.085686 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.085595    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a57950-a7f7-424c-8b42-542567916e3b-istiod-ca-cert\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:00.085686 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.085612    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-istio-data" (OuterVolumeSpecName: "istio-data") pod "05a57950-a7f7-424c-8b42-542567916e3b" (UID: "05a57950-a7f7-424c-8b42-542567916e3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:33:00.085686 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.085678    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "05a57950-a7f7-424c-8b42-542567916e3b" (UID: "05a57950-a7f7-424c-8b42-542567916e3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:33:00.087416 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.087394    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05a57950-a7f7-424c-8b42-542567916e3b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "05a57950-a7f7-424c-8b42-542567916e3b" (UID: "05a57950-a7f7-424c-8b42-542567916e3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:33:00.087795 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.087767    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a57950-a7f7-424c-8b42-542567916e3b-kube-api-access-lktbc" (OuterVolumeSpecName: "kube-api-access-lktbc") pod "05a57950-a7f7-424c-8b42-542567916e3b" (UID: "05a57950-a7f7-424c-8b42-542567916e3b"). InnerVolumeSpecName "kube-api-access-lktbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:33:00.087983 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.087963    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a57950-a7f7-424c-8b42-542567916e3b-istio-token" (OuterVolumeSpecName: "istio-token") pod "05a57950-a7f7-424c-8b42-542567916e3b" (UID: "05a57950-a7f7-424c-8b42-542567916e3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:33:00.087983 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.087965    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "05a57950-a7f7-424c-8b42-542567916e3b" (UID: "05a57950-a7f7-424c-8b42-542567916e3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:33:00.186293 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.186257    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a57950-a7f7-424c-8b42-542567916e3b-istio-token\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:00.186293 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.186284    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-istio-envoy\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:00.186293 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.186293    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a57950-a7f7-424c-8b42-542567916e3b-istio-podinfo\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:00.186293 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.186301    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lktbc\" (UniqueName: \"kubernetes.io/projected/05a57950-a7f7-424c-8b42-542567916e3b-kube-api-access-lktbc\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:00.186620 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.186311    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-workload-socket\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:00.186620 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.186319    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a57950-a7f7-424c-8b42-542567916e3b-istio-data\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:00.572438 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.572398    2554 generic.go:356] "Generic (PLEG): container finished" podID="05a57950-a7f7-424c-8b42-542567916e3b" containerID="887f49b4d9fa00ec044f267df5f3f01a6f202ff6ac50db6b69b45847b769de3f" exitCode=0
Sep 18 16:33:00.572653 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.572452    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"
Sep 18 16:33:00.572653 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.572483    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx" event={"ID":"05a57950-a7f7-424c-8b42-542567916e3b","Type":"ContainerDied","Data":"887f49b4d9fa00ec044f267df5f3f01a6f202ff6ac50db6b69b45847b769de3f"}
Sep 18 16:33:00.572653 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.572565    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx" event={"ID":"05a57950-a7f7-424c-8b42-542567916e3b","Type":"ContainerDied","Data":"dfe81a2f5a700a92d46252f3dff735e378cbd53ca68170bc23a670e2e2d289d7"}
Sep 18 16:33:00.572653 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.572589    2554 scope.go:117] "RemoveContainer" containerID="887f49b4d9fa00ec044f267df5f3f01a6f202ff6ac50db6b69b45847b769de3f"
Sep 18 16:33:00.582452 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.582434    2554 scope.go:117] "RemoveContainer" containerID="e6d21fe10ec4012b479c0381b484c0ccb1b518bfc8bd3693aff2f7a14576b639"
Sep 18 16:33:00.591250 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.591231    2554 scope.go:117] "RemoveContainer" containerID="887f49b4d9fa00ec044f267df5f3f01a6f202ff6ac50db6b69b45847b769de3f"
Sep 18 16:33:00.591516 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:33:00.591496    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887f49b4d9fa00ec044f267df5f3f01a6f202ff6ac50db6b69b45847b769de3f\": container with ID starting with 887f49b4d9fa00ec044f267df5f3f01a6f202ff6ac50db6b69b45847b769de3f not found: ID does not exist" containerID="887f49b4d9fa00ec044f267df5f3f01a6f202ff6ac50db6b69b45847b769de3f"
Sep 18 16:33:00.591584 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.591540    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887f49b4d9fa00ec044f267df5f3f01a6f202ff6ac50db6b69b45847b769de3f"} err="failed to get container status \"887f49b4d9fa00ec044f267df5f3f01a6f202ff6ac50db6b69b45847b769de3f\": rpc error: code = NotFound desc = could not find container \"887f49b4d9fa00ec044f267df5f3f01a6f202ff6ac50db6b69b45847b769de3f\": container with ID starting with 887f49b4d9fa00ec044f267df5f3f01a6f202ff6ac50db6b69b45847b769de3f not found: ID does not exist"
Sep 18 16:33:00.591584 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.591557    2554 scope.go:117] "RemoveContainer" containerID="e6d21fe10ec4012b479c0381b484c0ccb1b518bfc8bd3693aff2f7a14576b639"
Sep 18 16:33:00.591812 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:33:00.591795    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d21fe10ec4012b479c0381b484c0ccb1b518bfc8bd3693aff2f7a14576b639\": container with ID starting with e6d21fe10ec4012b479c0381b484c0ccb1b518bfc8bd3693aff2f7a14576b639 not found: ID does not exist" containerID="e6d21fe10ec4012b479c0381b484c0ccb1b518bfc8bd3693aff2f7a14576b639"
Sep 18 16:33:00.591862 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.591815    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d21fe10ec4012b479c0381b484c0ccb1b518bfc8bd3693aff2f7a14576b639"} err="failed to get container status \"e6d21fe10ec4012b479c0381b484c0ccb1b518bfc8bd3693aff2f7a14576b639\": rpc error: code = NotFound desc = could not find container \"e6d21fe10ec4012b479c0381b484c0ccb1b518bfc8bd3693aff2f7a14576b639\": container with ID starting with e6d21fe10ec4012b479c0381b484c0ccb1b518bfc8bd3693aff2f7a14576b639 not found: ID does not exist"
Sep 18 16:33:00.622321 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.622285    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"]
Sep 18 16:33:00.622481 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:00.622343    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ac5968-openshift-default-846b9b8856-cxxnx"]
Sep 18 16:33:01.278010 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:01.277976    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05a57950-a7f7-424c-8b42-542567916e3b" path="/var/lib/kubelet/pods/05a57950-a7f7-424c-8b42-542567916e3b/volumes"
Sep 18 16:33:11.000157 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.000125    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"]
Sep 18 16:33:11.000540 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.000380    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst" podUID="f35b8793-64a1-4d07-801e-f9aa31da7fc2" containerName="payload-pre-processing" containerID="cri-o://9a284650a7be00d49a723a5c8a5416dc56d06f3f9dd70da3c5f519f9f92131e0" gracePeriod=30
Sep 18 16:33:11.266329 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.266259    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"
Sep 18 16:33:11.290223 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.290175    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35b8793-64a1-4d07-801e-f9aa31da7fc2-plugins-config-volume\") pod \"f35b8793-64a1-4d07-801e-f9aa31da7fc2\" (UID: \"f35b8793-64a1-4d07-801e-f9aa31da7fc2\") "
Sep 18 16:33:11.290369 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.290298    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx6jn\" (UniqueName: \"kubernetes.io/projected/f35b8793-64a1-4d07-801e-f9aa31da7fc2-kube-api-access-sx6jn\") pod \"f35b8793-64a1-4d07-801e-f9aa31da7fc2\" (UID: \"f35b8793-64a1-4d07-801e-f9aa31da7fc2\") "
Sep 18 16:33:11.290571 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.290550    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f35b8793-64a1-4d07-801e-f9aa31da7fc2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f35b8793-64a1-4d07-801e-f9aa31da7fc2" (UID: "f35b8793-64a1-4d07-801e-f9aa31da7fc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:33:11.290714 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.290694    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35b8793-64a1-4d07-801e-f9aa31da7fc2-plugins-config-volume\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:11.292572 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.292552    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35b8793-64a1-4d07-801e-f9aa31da7fc2-kube-api-access-sx6jn" (OuterVolumeSpecName: "kube-api-access-sx6jn") pod "f35b8793-64a1-4d07-801e-f9aa31da7fc2" (UID: "f35b8793-64a1-4d07-801e-f9aa31da7fc2"). InnerVolumeSpecName "kube-api-access-sx6jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:33:11.392244 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.392211    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx6jn\" (UniqueName: \"kubernetes.io/projected/f35b8793-64a1-4d07-801e-f9aa31da7fc2-kube-api-access-sx6jn\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:11.631275 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.631178    2554 generic.go:356] "Generic (PLEG): container finished" podID="f35b8793-64a1-4d07-801e-f9aa31da7fc2" containerID="9a284650a7be00d49a723a5c8a5416dc56d06f3f9dd70da3c5f519f9f92131e0" exitCode=0
Sep 18 16:33:11.631275 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.631226    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"
Sep 18 16:33:11.631275 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.631263    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst" event={"ID":"f35b8793-64a1-4d07-801e-f9aa31da7fc2","Type":"ContainerDied","Data":"9a284650a7be00d49a723a5c8a5416dc56d06f3f9dd70da3c5f519f9f92131e0"}
Sep 18 16:33:11.631540 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.631327    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst" event={"ID":"f35b8793-64a1-4d07-801e-f9aa31da7fc2","Type":"ContainerDied","Data":"bedf4f2019a18f298cf6e8cacc7f335c1a7e97af70623a261ccfb2dc1946c940"}
Sep 18 16:33:11.631540 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.631352    2554 scope.go:117] "RemoveContainer" containerID="9a284650a7be00d49a723a5c8a5416dc56d06f3f9dd70da3c5f519f9f92131e0"
Sep 18 16:33:11.641148 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.641131    2554 scope.go:117] "RemoveContainer" containerID="9a284650a7be00d49a723a5c8a5416dc56d06f3f9dd70da3c5f519f9f92131e0"
Sep 18 16:33:11.641417 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:33:11.641398    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a284650a7be00d49a723a5c8a5416dc56d06f3f9dd70da3c5f519f9f92131e0\": container with ID starting with 9a284650a7be00d49a723a5c8a5416dc56d06f3f9dd70da3c5f519f9f92131e0 not found: ID does not exist" containerID="9a284650a7be00d49a723a5c8a5416dc56d06f3f9dd70da3c5f519f9f92131e0"
Sep 18 16:33:11.641463 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.641426    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a284650a7be00d49a723a5c8a5416dc56d06f3f9dd70da3c5f519f9f92131e0"} err="failed to get container status \"9a284650a7be00d49a723a5c8a5416dc56d06f3f9dd70da3c5f519f9f92131e0\": rpc error: code = NotFound desc = could not find container \"9a284650a7be00d49a723a5c8a5416dc56d06f3f9dd70da3c5f519f9f92131e0\": container with ID starting with 9a284650a7be00d49a723a5c8a5416dc56d06f3f9dd70da3c5f519f9f92131e0 not found: ID does not exist"
Sep 18 16:33:11.692377 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.692353    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"]
Sep 18 16:33:11.697815 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:11.697797    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9eae22-bfc97b5cd-bxtst"]
Sep 18 16:33:13.285336 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:13.285299    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f35b8793-64a1-4d07-801e-f9aa31da7fc2" path="/var/lib/kubelet/pods/f35b8793-64a1-4d07-801e-f9aa31da7fc2/volumes"
Sep 18 16:33:32.491653 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:32.491576    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"]
Sep 18 16:33:32.492189 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:32.491852    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6" podUID="52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb" containerName="maas-api" containerID="cri-o://813813d66f8e55396c124f3780c0541ff600b4c04a2f0f80cb3532cc2d425fa3" gracePeriod=30
Sep 18 16:33:32.658187 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:32.658151    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"]
Sep 18 16:33:32.658478 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:32.658400    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv" podUID="60ad33d7-6cf0-4bc7-ac01-423926cf2b1b" containerName="payload-pre-processing" containerID="cri-o://ded6034174fc1d4744284e101519885085cc689762ee8680b3d9e89bc937f7c0" gracePeriod=30
Sep 18 16:33:32.749121 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:32.749043    2554 generic.go:356] "Generic (PLEG): container finished" podID="52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb" containerID="813813d66f8e55396c124f3780c0541ff600b4c04a2f0f80cb3532cc2d425fa3" exitCode=0
Sep 18 16:33:32.749243 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:32.749120    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6" event={"ID":"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb","Type":"ContainerDied","Data":"813813d66f8e55396c124f3780c0541ff600b4c04a2f0f80cb3532cc2d425fa3"}
Sep 18 16:33:32.785298 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:32.785276    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:33:32.890736 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:32.890711    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"
Sep 18 16:33:32.899896 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:32.899871    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47mhr\" (UniqueName: \"kubernetes.io/projected/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-kube-api-access-47mhr\") pod \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\" (UID: \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\") "
Sep 18 16:33:32.900049 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:32.899932    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-metrics-certs\") pod \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\" (UID: \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\") "
Sep 18 16:33:32.900049 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:32.899954    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-openshift-service-ca\") pod \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\" (UID: \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\") "
Sep 18 16:33:32.900049 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:32.900001    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-maas-api-tls\") pod \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\" (UID: \"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb\") "
Sep 18 16:33:32.900359 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:32.900331    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb" (UID: "52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:33:32.902251 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:32.902223    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb" (UID: "52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:33:32.902251 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:32.902223    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-kube-api-access-47mhr" (OuterVolumeSpecName: "kube-api-access-47mhr") pod "52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb" (UID: "52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb"). InnerVolumeSpecName "kube-api-access-47mhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:33:32.902251 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:32.902240    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb" (UID: "52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:33:33.000640 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.000547    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ad33d7-6cf0-4bc7-ac01-423926cf2b1b-plugins-config-volume\") pod \"60ad33d7-6cf0-4bc7-ac01-423926cf2b1b\" (UID: \"60ad33d7-6cf0-4bc7-ac01-423926cf2b1b\") "
Sep 18 16:33:33.000794 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.000683    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh6t5\" (UniqueName: \"kubernetes.io/projected/60ad33d7-6cf0-4bc7-ac01-423926cf2b1b-kube-api-access-lh6t5\") pod \"60ad33d7-6cf0-4bc7-ac01-423926cf2b1b\" (UID: \"60ad33d7-6cf0-4bc7-ac01-423926cf2b1b\") "
Sep 18 16:33:33.000918 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.000906    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47mhr\" (UniqueName: \"kubernetes.io/projected/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-kube-api-access-47mhr\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:33.000961 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.000919    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-metrics-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:33.000961 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.000930    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-openshift-service-ca\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:33.000961 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.000942    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb-maas-api-tls\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:33.001056 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.001008    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ad33d7-6cf0-4bc7-ac01-423926cf2b1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60ad33d7-6cf0-4bc7-ac01-423926cf2b1b" (UID: "60ad33d7-6cf0-4bc7-ac01-423926cf2b1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:33:33.002908 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.002886    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ad33d7-6cf0-4bc7-ac01-423926cf2b1b-kube-api-access-lh6t5" (OuterVolumeSpecName: "kube-api-access-lh6t5") pod "60ad33d7-6cf0-4bc7-ac01-423926cf2b1b" (UID: "60ad33d7-6cf0-4bc7-ac01-423926cf2b1b"). InnerVolumeSpecName "kube-api-access-lh6t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:33:33.101544 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.101486    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ad33d7-6cf0-4bc7-ac01-423926cf2b1b-plugins-config-volume\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:33.101544 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.101517    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh6t5\" (UniqueName: \"kubernetes.io/projected/60ad33d7-6cf0-4bc7-ac01-423926cf2b1b-kube-api-access-lh6t5\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:33.756378 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.756341    2554 generic.go:356] "Generic (PLEG): container finished" podID="60ad33d7-6cf0-4bc7-ac01-423926cf2b1b" containerID="ded6034174fc1d4744284e101519885085cc689762ee8680b3d9e89bc937f7c0" exitCode=0
Sep 18 16:33:33.756881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.756385    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"
Sep 18 16:33:33.756881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.756432    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv" event={"ID":"60ad33d7-6cf0-4bc7-ac01-423926cf2b1b","Type":"ContainerDied","Data":"ded6034174fc1d4744284e101519885085cc689762ee8680b3d9e89bc937f7c0"}
Sep 18 16:33:33.756881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.756483    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv" event={"ID":"60ad33d7-6cf0-4bc7-ac01-423926cf2b1b","Type":"ContainerDied","Data":"9c3154846c4c8a63cf86f840872fca30847cd465a245d100a784864180d84630"}
Sep 18 16:33:33.756881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.756505    2554 scope.go:117] "RemoveContainer" containerID="ded6034174fc1d4744284e101519885085cc689762ee8680b3d9e89bc937f7c0"
Sep 18 16:33:33.757995 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.757978    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"
Sep 18 16:33:33.758095 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.757984    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6" event={"ID":"52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb","Type":"ContainerDied","Data":"0114183c643924ef2c3ff23f13433745c3759a32c313c7078963f4b338d0dd78"}
Sep 18 16:33:33.765694 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.765675    2554 scope.go:117] "RemoveContainer" containerID="ded6034174fc1d4744284e101519885085cc689762ee8680b3d9e89bc937f7c0"
Sep 18 16:33:33.765978 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:33:33.765960    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded6034174fc1d4744284e101519885085cc689762ee8680b3d9e89bc937f7c0\": container with ID starting with ded6034174fc1d4744284e101519885085cc689762ee8680b3d9e89bc937f7c0 not found: ID does not exist" containerID="ded6034174fc1d4744284e101519885085cc689762ee8680b3d9e89bc937f7c0"
Sep 18 16:33:33.766038 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.765986    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded6034174fc1d4744284e101519885085cc689762ee8680b3d9e89bc937f7c0"} err="failed to get container status \"ded6034174fc1d4744284e101519885085cc689762ee8680b3d9e89bc937f7c0\": rpc error: code = NotFound desc = could not find container \"ded6034174fc1d4744284e101519885085cc689762ee8680b3d9e89bc937f7c0\": container with ID starting with ded6034174fc1d4744284e101519885085cc689762ee8680b3d9e89bc937f7c0 not found: ID does not exist"
Sep 18 16:33:33.766038 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.766004    2554 scope.go:117] "RemoveContainer" containerID="813813d66f8e55396c124f3780c0541ff600b4c04a2f0f80cb3532cc2d425fa3"
Sep 18 16:33:33.780133 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.780103    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"]
Sep 18 16:33:33.804435 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.804411    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5b98dc-85898b8c74-fjbr6"]
Sep 18 16:33:33.822632 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.822605    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"]
Sep 18 16:33:33.831196 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:33.831168    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5b98dc-85d9f6d54b-vqhqv"]
Sep 18 16:33:35.277733 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:35.277701    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb" path="/var/lib/kubelet/pods/52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb/volumes"
Sep 18 16:33:35.278119 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:35.278079    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60ad33d7-6cf0-4bc7-ac01-423926cf2b1b" path="/var/lib/kubelet/pods/60ad33d7-6cf0-4bc7-ac01-423926cf2b1b/volumes"
Sep 18 16:33:41.900903 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:41.900861    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"]
Sep 18 16:33:41.901382 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:41.901111    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z" podUID="ecb6fd0b-b70a-4e35-804f-030e67d4bf69" containerName="istio-proxy" containerID="cri-o://ff38be622c4fd7fac3c92e23de18d629f31e40a99a054083ef668b7bd2979bb8" gracePeriod=30
Sep 18 16:33:47.151079 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.151054    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:33:47.243347 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.243316    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-data\") pod \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") "
Sep 18 16:33:47.243516 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.243357    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-credential-socket\") pod \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") "
Sep 18 16:33:47.243516 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.243381    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-workload-socket\") pod \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") "
Sep 18 16:33:47.243516 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.243410    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istiod-ca-cert\") pod \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") "
Sep 18 16:33:47.243700 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.243562    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-envoy\") pod \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") "
Sep 18 16:33:47.243700 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.243624    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-data" (OuterVolumeSpecName: "istio-data") pod "ecb6fd0b-b70a-4e35-804f-030e67d4bf69" (UID: "ecb6fd0b-b70a-4e35-804f-030e67d4bf69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:33:47.243700 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.243643    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-workload-certs\") pod \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") "
Sep 18 16:33:47.243700 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.243673    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-token\") pod \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") "
Sep 18 16:33:47.243700 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.243653    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ecb6fd0b-b70a-4e35-804f-030e67d4bf69" (UID: "ecb6fd0b-b70a-4e35-804f-030e67d4bf69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:33:47.243700 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.243693    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ecb6fd0b-b70a-4e35-804f-030e67d4bf69" (UID: "ecb6fd0b-b70a-4e35-804f-030e67d4bf69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:33:47.244043 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.243711    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-podinfo\") pod \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") "
Sep 18 16:33:47.244043 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.243747    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfc5l\" (UniqueName: \"kubernetes.io/projected/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-kube-api-access-vfc5l\") pod \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\" (UID: \"ecb6fd0b-b70a-4e35-804f-030e67d4bf69\") "
Sep 18 16:33:47.244043 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.243828    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ecb6fd0b-b70a-4e35-804f-030e67d4bf69" (UID: "ecb6fd0b-b70a-4e35-804f-030e67d4bf69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:33:47.244043 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.243828    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ecb6fd0b-b70a-4e35-804f-030e67d4bf69" (UID: "ecb6fd0b-b70a-4e35-804f-030e67d4bf69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:33:47.244180 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.244080    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-data\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:47.244180 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.244100    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-credential-socket\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:47.244180 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.244112    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-workload-socket\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:47.244180 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.244125    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istiod-ca-cert\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:47.244180 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.244139    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-workload-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:47.245906 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.245877    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ecb6fd0b-b70a-4e35-804f-030e67d4bf69" (UID: "ecb6fd0b-b70a-4e35-804f-030e67d4bf69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:33:47.246030 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.245935    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-token" (OuterVolumeSpecName: "istio-token") pod "ecb6fd0b-b70a-4e35-804f-030e67d4bf69" (UID: "ecb6fd0b-b70a-4e35-804f-030e67d4bf69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:33:47.246327 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.246301    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ecb6fd0b-b70a-4e35-804f-030e67d4bf69" (UID: "ecb6fd0b-b70a-4e35-804f-030e67d4bf69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:33:47.246421 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.246340    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-kube-api-access-vfc5l" (OuterVolumeSpecName: "kube-api-access-vfc5l") pod "ecb6fd0b-b70a-4e35-804f-030e67d4bf69" (UID: "ecb6fd0b-b70a-4e35-804f-030e67d4bf69"). InnerVolumeSpecName "kube-api-access-vfc5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:33:47.345210 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.345184    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-envoy\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:47.345210 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.345212    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-token\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:47.345386 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.345225    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-istio-podinfo\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:47.345386 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.345237    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfc5l\" (UniqueName: \"kubernetes.io/projected/ecb6fd0b-b70a-4e35-804f-030e67d4bf69-kube-api-access-vfc5l\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:47.836928 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.836895    2554 generic.go:356] "Generic (PLEG): container finished" podID="ecb6fd0b-b70a-4e35-804f-030e67d4bf69" containerID="ff38be622c4fd7fac3c92e23de18d629f31e40a99a054083ef668b7bd2979bb8" exitCode=0
Sep 18 16:33:47.837186 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.836943    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"
Sep 18 16:33:47.837186 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.836958    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z" event={"ID":"ecb6fd0b-b70a-4e35-804f-030e67d4bf69","Type":"ContainerDied","Data":"ff38be622c4fd7fac3c92e23de18d629f31e40a99a054083ef668b7bd2979bb8"}
Sep 18 16:33:47.837186 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.837000    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z" event={"ID":"ecb6fd0b-b70a-4e35-804f-030e67d4bf69","Type":"ContainerDied","Data":"5bcb95906b5450da741e752ad7ee4d75f645c2931d151b3040caa11ccf57d14e"}
Sep 18 16:33:47.837186 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.837023    2554 scope.go:117] "RemoveContainer" containerID="ff38be622c4fd7fac3c92e23de18d629f31e40a99a054083ef668b7bd2979bb8"
Sep 18 16:33:47.845985 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.845954    2554 scope.go:117] "RemoveContainer" containerID="4deba3560df69298d62c2368ddd72b87aa7da5f07f807225517b067c5f3bc696"
Sep 18 16:33:47.854097 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.854075    2554 scope.go:117] "RemoveContainer" containerID="ff38be622c4fd7fac3c92e23de18d629f31e40a99a054083ef668b7bd2979bb8"
Sep 18 16:33:47.854360 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:33:47.854342    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff38be622c4fd7fac3c92e23de18d629f31e40a99a054083ef668b7bd2979bb8\": container with ID starting with ff38be622c4fd7fac3c92e23de18d629f31e40a99a054083ef668b7bd2979bb8 not found: ID does not exist" containerID="ff38be622c4fd7fac3c92e23de18d629f31e40a99a054083ef668b7bd2979bb8"
Sep 18 16:33:47.854420 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.854368    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff38be622c4fd7fac3c92e23de18d629f31e40a99a054083ef668b7bd2979bb8"} err="failed to get container status \"ff38be622c4fd7fac3c92e23de18d629f31e40a99a054083ef668b7bd2979bb8\": rpc error: code = NotFound desc = could not find container \"ff38be622c4fd7fac3c92e23de18d629f31e40a99a054083ef668b7bd2979bb8\": container with ID starting with ff38be622c4fd7fac3c92e23de18d629f31e40a99a054083ef668b7bd2979bb8 not found: ID does not exist"
Sep 18 16:33:47.854420 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.854385    2554 scope.go:117] "RemoveContainer" containerID="4deba3560df69298d62c2368ddd72b87aa7da5f07f807225517b067c5f3bc696"
Sep 18 16:33:47.854651 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:33:47.854635    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4deba3560df69298d62c2368ddd72b87aa7da5f07f807225517b067c5f3bc696\": container with ID starting with 4deba3560df69298d62c2368ddd72b87aa7da5f07f807225517b067c5f3bc696 not found: ID does not exist" containerID="4deba3560df69298d62c2368ddd72b87aa7da5f07f807225517b067c5f3bc696"
Sep 18 16:33:47.854708 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.854656    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4deba3560df69298d62c2368ddd72b87aa7da5f07f807225517b067c5f3bc696"} err="failed to get container status \"4deba3560df69298d62c2368ddd72b87aa7da5f07f807225517b067c5f3bc696\": rpc error: code = NotFound desc = could not find container \"4deba3560df69298d62c2368ddd72b87aa7da5f07f807225517b067c5f3bc696\": container with ID starting with 4deba3560df69298d62c2368ddd72b87aa7da5f07f807225517b067c5f3bc696 not found: ID does not exist"
Sep 18 16:33:47.867084 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.867062    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"]
Sep 18 16:33:47.873806 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:47.873780    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5b98dc-openshift-default-7d5d44d976-fr42z"]
Sep 18 16:33:49.278007 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:49.277977    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecb6fd0b-b70a-4e35-804f-030e67d4bf69" path="/var/lib/kubelet/pods/ecb6fd0b-b70a-4e35-804f-030e67d4bf69/volumes"
Sep 18 16:33:53.179038 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.179007    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"]
Sep 18 16:33:53.179511 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.179276    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk" podUID="58dbe8aa-7537-4391-8486-981345f7c0b5" containerName="payload-pre-processing" containerID="cri-o://d1bfff5be14a53d7c2c17331d5bb79ed00ccb9bf0eb17012038f39c93118707b" gracePeriod=30
Sep 18 16:33:53.428363 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.428338    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"
Sep 18 16:33:53.507989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.507882    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58dbe8aa-7537-4391-8486-981345f7c0b5-plugins-config-volume\") pod \"58dbe8aa-7537-4391-8486-981345f7c0b5\" (UID: \"58dbe8aa-7537-4391-8486-981345f7c0b5\") "
Sep 18 16:33:53.507989 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.507970    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjjkl\" (UniqueName: \"kubernetes.io/projected/58dbe8aa-7537-4391-8486-981345f7c0b5-kube-api-access-fjjkl\") pod \"58dbe8aa-7537-4391-8486-981345f7c0b5\" (UID: \"58dbe8aa-7537-4391-8486-981345f7c0b5\") "
Sep 18 16:33:53.508370 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.508334    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58dbe8aa-7537-4391-8486-981345f7c0b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "58dbe8aa-7537-4391-8486-981345f7c0b5" (UID: "58dbe8aa-7537-4391-8486-981345f7c0b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:33:53.510780 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.510752    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58dbe8aa-7537-4391-8486-981345f7c0b5-kube-api-access-fjjkl" (OuterVolumeSpecName: "kube-api-access-fjjkl") pod "58dbe8aa-7537-4391-8486-981345f7c0b5" (UID: "58dbe8aa-7537-4391-8486-981345f7c0b5"). InnerVolumeSpecName "kube-api-access-fjjkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:33:53.609198 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.609163    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjjkl\" (UniqueName: \"kubernetes.io/projected/58dbe8aa-7537-4391-8486-981345f7c0b5-kube-api-access-fjjkl\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:53.609198 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.609192    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58dbe8aa-7537-4391-8486-981345f7c0b5-plugins-config-volume\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:53.871508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.871426    2554 generic.go:356] "Generic (PLEG): container finished" podID="58dbe8aa-7537-4391-8486-981345f7c0b5" containerID="d1bfff5be14a53d7c2c17331d5bb79ed00ccb9bf0eb17012038f39c93118707b" exitCode=0
Sep 18 16:33:53.871508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.871462    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"
Sep 18 16:33:53.871718 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.871516    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk" event={"ID":"58dbe8aa-7537-4391-8486-981345f7c0b5","Type":"ContainerDied","Data":"d1bfff5be14a53d7c2c17331d5bb79ed00ccb9bf0eb17012038f39c93118707b"}
Sep 18 16:33:53.871718 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.871581    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk" event={"ID":"58dbe8aa-7537-4391-8486-981345f7c0b5","Type":"ContainerDied","Data":"f1c875d24df5547c788afcdbc08282b6a433e94a683a2a4a5b1b06b5d8ffaecd"}
Sep 18 16:33:53.871718 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.871605    2554 scope.go:117] "RemoveContainer" containerID="d1bfff5be14a53d7c2c17331d5bb79ed00ccb9bf0eb17012038f39c93118707b"
Sep 18 16:33:53.881361 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.881345    2554 scope.go:117] "RemoveContainer" containerID="d1bfff5be14a53d7c2c17331d5bb79ed00ccb9bf0eb17012038f39c93118707b"
Sep 18 16:33:53.881658 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:33:53.881638    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1bfff5be14a53d7c2c17331d5bb79ed00ccb9bf0eb17012038f39c93118707b\": container with ID starting with d1bfff5be14a53d7c2c17331d5bb79ed00ccb9bf0eb17012038f39c93118707b not found: ID does not exist" containerID="d1bfff5be14a53d7c2c17331d5bb79ed00ccb9bf0eb17012038f39c93118707b"
Sep 18 16:33:53.881718 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.881666    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bfff5be14a53d7c2c17331d5bb79ed00ccb9bf0eb17012038f39c93118707b"} err="failed to get container status \"d1bfff5be14a53d7c2c17331d5bb79ed00ccb9bf0eb17012038f39c93118707b\": rpc error: code = NotFound desc = could not find container \"d1bfff5be14a53d7c2c17331d5bb79ed00ccb9bf0eb17012038f39c93118707b\": container with ID starting with d1bfff5be14a53d7c2c17331d5bb79ed00ccb9bf0eb17012038f39c93118707b not found: ID does not exist"
Sep 18 16:33:53.898997 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.898977    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"]
Sep 18 16:33:53.913334 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:53.913312    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5865ea-fb7d99446-hxpxk"]
Sep 18 16:33:55.278158 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.278115    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58dbe8aa-7537-4391-8486-981345f7c0b5" path="/var/lib/kubelet/pods/58dbe8aa-7537-4391-8486-981345f7c0b5/volumes"
Sep 18 16:33:55.501199 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.501166    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"]
Sep 18 16:33:55.501418 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.501391    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56" podUID="fbebcbf4-fa27-4811-9cad-5ab137b70841" containerName="maas-api" containerID="cri-o://ffc7a7a62fb1f68afd277de6d7b11bd82835751d4ffc4b94bb3f91726739afee" gracePeriod=30
Sep 18 16:33:55.583592 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.583495    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1eec1d-29829150-g2qjw"]
Sep 18 16:33:55.588719 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.588695    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1eec1d-29829150-g2qjw"]
Sep 18 16:33:55.765980 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.765908    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:33:55.829831 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.829793    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbebcbf4-fa27-4811-9cad-5ab137b70841-metrics-certs\") pod \"fbebcbf4-fa27-4811-9cad-5ab137b70841\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") "
Sep 18 16:33:55.830007 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.829847    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbebcbf4-fa27-4811-9cad-5ab137b70841-maas-api-tls\") pod \"fbebcbf4-fa27-4811-9cad-5ab137b70841\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") "
Sep 18 16:33:55.830007 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.829915    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc8bd\" (UniqueName: \"kubernetes.io/projected/fbebcbf4-fa27-4811-9cad-5ab137b70841-kube-api-access-mc8bd\") pod \"fbebcbf4-fa27-4811-9cad-5ab137b70841\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") "
Sep 18 16:33:55.830007 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.829938    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbebcbf4-fa27-4811-9cad-5ab137b70841-openshift-service-ca\") pod \"fbebcbf4-fa27-4811-9cad-5ab137b70841\" (UID: \"fbebcbf4-fa27-4811-9cad-5ab137b70841\") "
Sep 18 16:33:55.830373 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.830342    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbebcbf4-fa27-4811-9cad-5ab137b70841-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fbebcbf4-fa27-4811-9cad-5ab137b70841" (UID: "fbebcbf4-fa27-4811-9cad-5ab137b70841"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:33:55.832202 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.832173    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbebcbf4-fa27-4811-9cad-5ab137b70841-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fbebcbf4-fa27-4811-9cad-5ab137b70841" (UID: "fbebcbf4-fa27-4811-9cad-5ab137b70841"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:33:55.832202 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.832189    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbebcbf4-fa27-4811-9cad-5ab137b70841-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fbebcbf4-fa27-4811-9cad-5ab137b70841" (UID: "fbebcbf4-fa27-4811-9cad-5ab137b70841"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:33:55.832341 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.832228    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbebcbf4-fa27-4811-9cad-5ab137b70841-kube-api-access-mc8bd" (OuterVolumeSpecName: "kube-api-access-mc8bd") pod "fbebcbf4-fa27-4811-9cad-5ab137b70841" (UID: "fbebcbf4-fa27-4811-9cad-5ab137b70841"). InnerVolumeSpecName "kube-api-access-mc8bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:33:55.884695 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.884657    2554 generic.go:356] "Generic (PLEG): container finished" podID="fbebcbf4-fa27-4811-9cad-5ab137b70841" containerID="ffc7a7a62fb1f68afd277de6d7b11bd82835751d4ffc4b94bb3f91726739afee" exitCode=0
Sep 18 16:33:55.884856 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.884709    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"
Sep 18 16:33:55.884856 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.884737    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56" event={"ID":"fbebcbf4-fa27-4811-9cad-5ab137b70841","Type":"ContainerDied","Data":"ffc7a7a62fb1f68afd277de6d7b11bd82835751d4ffc4b94bb3f91726739afee"}
Sep 18 16:33:55.884856 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.884788    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56" event={"ID":"fbebcbf4-fa27-4811-9cad-5ab137b70841","Type":"ContainerDied","Data":"8c7a44a6a7862d0ddada5d3c3e27e78165c94ba88efec2329e8a7737978ac298"}
Sep 18 16:33:55.884856 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.884820    2554 scope.go:117] "RemoveContainer" containerID="ffc7a7a62fb1f68afd277de6d7b11bd82835751d4ffc4b94bb3f91726739afee"
Sep 18 16:33:55.894429 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.894410    2554 scope.go:117] "RemoveContainer" containerID="ffc7a7a62fb1f68afd277de6d7b11bd82835751d4ffc4b94bb3f91726739afee"
Sep 18 16:33:55.894719 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:33:55.894697    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffc7a7a62fb1f68afd277de6d7b11bd82835751d4ffc4b94bb3f91726739afee\": container with ID starting with ffc7a7a62fb1f68afd277de6d7b11bd82835751d4ffc4b94bb3f91726739afee not found: ID does not exist" containerID="ffc7a7a62fb1f68afd277de6d7b11bd82835751d4ffc4b94bb3f91726739afee"
Sep 18 16:33:55.894776 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.894726    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffc7a7a62fb1f68afd277de6d7b11bd82835751d4ffc4b94bb3f91726739afee"} err="failed to get container status \"ffc7a7a62fb1f68afd277de6d7b11bd82835751d4ffc4b94bb3f91726739afee\": rpc error: code = NotFound desc = could not find container \"ffc7a7a62fb1f68afd277de6d7b11bd82835751d4ffc4b94bb3f91726739afee\": container with ID starting with ffc7a7a62fb1f68afd277de6d7b11bd82835751d4ffc4b94bb3f91726739afee not found: ID does not exist"
Sep 18 16:33:55.925516 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.925487    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"]
Sep 18 16:33:55.931472 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.931445    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbebcbf4-fa27-4811-9cad-5ab137b70841-metrics-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:55.931472 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.931469    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbebcbf4-fa27-4811-9cad-5ab137b70841-maas-api-tls\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:55.931472 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.931478    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc8bd\" (UniqueName: \"kubernetes.io/projected/fbebcbf4-fa27-4811-9cad-5ab137b70841-kube-api-access-mc8bd\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:55.931712 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.931488    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbebcbf4-fa27-4811-9cad-5ab137b70841-openshift-service-ca\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:33:55.940331 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:55.940299    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1eec1d-6f97dc4f7f-9ln56"]
Sep 18 16:33:57.278106 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:57.278069    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f246fb96-3d1d-4003-ba89-5441d87a7094" path="/var/lib/kubelet/pods/f246fb96-3d1d-4003-ba89-5441d87a7094/volumes"
Sep 18 16:33:57.278494 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:33:57.278422    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbebcbf4-fa27-4811-9cad-5ab137b70841" path="/var/lib/kubelet/pods/fbebcbf4-fa27-4811-9cad-5ab137b70841/volumes"
Sep 18 16:34:01.528590 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:01.528550    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"]
Sep 18 16:34:01.529021 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:01.528756    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq" podUID="9953f9be-cbc3-44a4-a29d-4a301e468844" containerName="istio-proxy" containerID="cri-o://3bc2c66dbda8ff393b938c13578d72d02282d4e3f88302a68da4e4be640e3fc5" gracePeriod=30
Sep 18 16:34:03.120583 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:03.120548    2554 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 16:34:03.121054 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:03.120610    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq" podUID="9953f9be-cbc3-44a4-a29d-4a301e468844" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:34:06.794035 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.793970    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:34:06.943558 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.942728    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-data\") pod \"9953f9be-cbc3-44a4-a29d-4a301e468844\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") "
Sep 18 16:34:06.943558 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.942806    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-workload-certs\") pod \"9953f9be-cbc3-44a4-a29d-4a301e468844\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") "
Sep 18 16:34:06.943558 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.942841    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-envoy\") pod \"9953f9be-cbc3-44a4-a29d-4a301e468844\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") "
Sep 18 16:34:06.943558 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.942884    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-workload-socket\") pod \"9953f9be-cbc3-44a4-a29d-4a301e468844\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") "
Sep 18 16:34:06.943558 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.942916    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj4bt\" (UniqueName: \"kubernetes.io/projected/9953f9be-cbc3-44a4-a29d-4a301e468844-kube-api-access-cj4bt\") pod \"9953f9be-cbc3-44a4-a29d-4a301e468844\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") "
Sep 18 16:34:06.943558 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.942982    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-podinfo\") pod \"9953f9be-cbc3-44a4-a29d-4a301e468844\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") "
Sep 18 16:34:06.943558 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.943021    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9953f9be-cbc3-44a4-a29d-4a301e468844-istiod-ca-cert\") pod \"9953f9be-cbc3-44a4-a29d-4a301e468844\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") "
Sep 18 16:34:06.943558 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.943077    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-credential-socket\") pod \"9953f9be-cbc3-44a4-a29d-4a301e468844\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") "
Sep 18 16:34:06.943558 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.943143    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-token\") pod \"9953f9be-cbc3-44a4-a29d-4a301e468844\" (UID: \"9953f9be-cbc3-44a4-a29d-4a301e468844\") "
Sep 18 16:34:06.943558 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.943458    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9953f9be-cbc3-44a4-a29d-4a301e468844" (UID: "9953f9be-cbc3-44a4-a29d-4a301e468844"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:06.944182 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.943782    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-workload-socket\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:34:06.944182 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.943985    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-data" (OuterVolumeSpecName: "istio-data") pod "9953f9be-cbc3-44a4-a29d-4a301e468844" (UID: "9953f9be-cbc3-44a4-a29d-4a301e468844"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:06.944182 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.944154    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9953f9be-cbc3-44a4-a29d-4a301e468844" (UID: "9953f9be-cbc3-44a4-a29d-4a301e468844"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:06.947436 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.947007    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9953f9be-cbc3-44a4-a29d-4a301e468844" (UID: "9953f9be-cbc3-44a4-a29d-4a301e468844"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:06.947436 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.947331    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9953f9be-cbc3-44a4-a29d-4a301e468844-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9953f9be-cbc3-44a4-a29d-4a301e468844" (UID: "9953f9be-cbc3-44a4-a29d-4a301e468844"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:34:06.949641 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.949617    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"
Sep 18 16:34:06.949927 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.949628    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9953f9be-cbc3-44a4-a29d-4a301e468844" (UID: "9953f9be-cbc3-44a4-a29d-4a301e468844"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:06.949927 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.949844    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-token" (OuterVolumeSpecName: "istio-token") pod "9953f9be-cbc3-44a4-a29d-4a301e468844" (UID: "9953f9be-cbc3-44a4-a29d-4a301e468844"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:06.949927 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.949912    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9953f9be-cbc3-44a4-a29d-4a301e468844-kube-api-access-cj4bt" (OuterVolumeSpecName: "kube-api-access-cj4bt") pod "9953f9be-cbc3-44a4-a29d-4a301e468844" (UID: "9953f9be-cbc3-44a4-a29d-4a301e468844"). InnerVolumeSpecName "kube-api-access-cj4bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:06.950159 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.949651    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq" event={"ID":"9953f9be-cbc3-44a4-a29d-4a301e468844","Type":"ContainerDied","Data":"3bc2c66dbda8ff393b938c13578d72d02282d4e3f88302a68da4e4be640e3fc5"}
Sep 18 16:34:06.950159 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.950025    2554 scope.go:117] "RemoveContainer" containerID="3bc2c66dbda8ff393b938c13578d72d02282d4e3f88302a68da4e4be640e3fc5"
Sep 18 16:34:06.950159 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.949490    2554 generic.go:356] "Generic (PLEG): container finished" podID="9953f9be-cbc3-44a4-a29d-4a301e468844" containerID="3bc2c66dbda8ff393b938c13578d72d02282d4e3f88302a68da4e4be640e3fc5" exitCode=0
Sep 18 16:34:06.950159 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.950113    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9953f9be-cbc3-44a4-a29d-4a301e468844" (UID: "9953f9be-cbc3-44a4-a29d-4a301e468844"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:34:06.950159 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.950125    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq" event={"ID":"9953f9be-cbc3-44a4-a29d-4a301e468844","Type":"ContainerDied","Data":"6d558da3aa3211f483e38ca64a7bed77ddb4794f037540f81bb71ac40d51aa14"}
Sep 18 16:34:06.967961 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.967935    2554 scope.go:117] "RemoveContainer" containerID="9922900e3a276845c17395b8124e886e7da75853f80028d14e9e42929cf473ab"
Sep 18 16:34:06.977361 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.977332    2554 scope.go:117] "RemoveContainer" containerID="3bc2c66dbda8ff393b938c13578d72d02282d4e3f88302a68da4e4be640e3fc5"
Sep 18 16:34:06.978721 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:34:06.978690    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bc2c66dbda8ff393b938c13578d72d02282d4e3f88302a68da4e4be640e3fc5\": container with ID starting with 3bc2c66dbda8ff393b938c13578d72d02282d4e3f88302a68da4e4be640e3fc5 not found: ID does not exist" containerID="3bc2c66dbda8ff393b938c13578d72d02282d4e3f88302a68da4e4be640e3fc5"
Sep 18 16:34:06.978836 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.978735    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bc2c66dbda8ff393b938c13578d72d02282d4e3f88302a68da4e4be640e3fc5"} err="failed to get container status \"3bc2c66dbda8ff393b938c13578d72d02282d4e3f88302a68da4e4be640e3fc5\": rpc error: code = NotFound desc = could not find container \"3bc2c66dbda8ff393b938c13578d72d02282d4e3f88302a68da4e4be640e3fc5\": container with ID starting with 3bc2c66dbda8ff393b938c13578d72d02282d4e3f88302a68da4e4be640e3fc5 not found: ID does not exist"
Sep 18 16:34:06.978836 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.978757    2554 scope.go:117] "RemoveContainer" containerID="9922900e3a276845c17395b8124e886e7da75853f80028d14e9e42929cf473ab"
Sep 18 16:34:06.979208 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:34:06.979184    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9922900e3a276845c17395b8124e886e7da75853f80028d14e9e42929cf473ab\": container with ID starting with 9922900e3a276845c17395b8124e886e7da75853f80028d14e9e42929cf473ab not found: ID does not exist" containerID="9922900e3a276845c17395b8124e886e7da75853f80028d14e9e42929cf473ab"
Sep 18 16:34:06.979300 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:06.979210    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9922900e3a276845c17395b8124e886e7da75853f80028d14e9e42929cf473ab"} err="failed to get container status \"9922900e3a276845c17395b8124e886e7da75853f80028d14e9e42929cf473ab\": rpc error: code = NotFound desc = could not find container \"9922900e3a276845c17395b8124e886e7da75853f80028d14e9e42929cf473ab\": container with ID starting with 9922900e3a276845c17395b8124e886e7da75853f80028d14e9e42929cf473ab not found: ID does not exist"
Sep 18 16:34:07.044884 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:07.044785    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj4bt\" (UniqueName: \"kubernetes.io/projected/9953f9be-cbc3-44a4-a29d-4a301e468844-kube-api-access-cj4bt\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:34:07.044884 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:07.044821    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-podinfo\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:34:07.044884 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:07.044834    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9953f9be-cbc3-44a4-a29d-4a301e468844-istiod-ca-cert\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:34:07.044884 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:07.044845    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-credential-socket\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:34:07.044884 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:07.044857    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-token\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:34:07.044884 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:07.044867    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-data\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:34:07.044884 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:07.044878    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-workload-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:34:07.044884 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:07.044888    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9953f9be-cbc3-44a4-a29d-4a301e468844-istio-envoy\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:34:07.278874 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:07.278846    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"]
Sep 18 16:34:07.283144 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:07.283117    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5865ea-openshift-default-679dc84496-xwlgq"]
Sep 18 16:34:09.279101 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:34:09.279070    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9953f9be-cbc3-44a4-a29d-4a301e468844" path="/var/lib/kubelet/pods/9953f9be-cbc3-44a4-a29d-4a301e468844/volumes"
Sep 18 16:35:41.241849 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:41.241812    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"]
Sep 18 16:35:42.358477 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:42.358438    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"]
Sep 18 16:35:56.261393 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.261328    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466" podUID="fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8" containerName="main" containerID="cri-o://6fcb776b48605aa1182226f1b4f4e8607567176639396824dfe6e3110388249a" gracePeriod=45
Sep 18 16:35:56.498898 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.498861    2554 generic.go:356] "Generic (PLEG): container finished" podID="fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8" containerID="6fcb776b48605aa1182226f1b4f4e8607567176639396824dfe6e3110388249a" exitCode=0
Sep 18 16:35:56.499023 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.498928    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466" event={"ID":"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8","Type":"ContainerDied","Data":"6fcb776b48605aa1182226f1b4f4e8607567176639396824dfe6e3110388249a"}
Sep 18 16:35:56.515143 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.515088    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:35:56.639793 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.639765    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-model-cache\") pod \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") "
Sep 18 16:35:56.639793 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.639797    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-tls-certs\") pod \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") "
Sep 18 16:35:56.640032 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.639821    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-tmp-dir\") pod \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") "
Sep 18 16:35:56.640032 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.639843    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-dshm\") pod \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") "
Sep 18 16:35:56.640032 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.639932    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk8lv\" (UniqueName: \"kubernetes.io/projected/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-kube-api-access-lk8lv\") pod \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") "
Sep 18 16:35:56.640032 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.639997    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-home\") pod \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\" (UID: \"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8\") "
Sep 18 16:35:56.640263 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.640072    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8" (UID: "fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:35:56.640263 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.640075    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8" (UID: "fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:35:56.640263 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.640244    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-home" (OuterVolumeSpecName: "home") pod "fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8" (UID: "fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:35:56.640437 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.640334    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-model-cache\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:35:56.640437 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.640353    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-tmp-dir\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:35:56.640437 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.640367    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-home\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:35:56.642196 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.642171    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-dshm" (OuterVolumeSpecName: "dshm") pod "fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8" (UID: "fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:35:56.642299 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.642221    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8" (UID: "fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:35:56.642299 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.642248    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-kube-api-access-lk8lv" (OuterVolumeSpecName: "kube-api-access-lk8lv") pod "fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8" (UID: "fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8"). InnerVolumeSpecName "kube-api-access-lk8lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:35:56.741315 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.741279    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk8lv\" (UniqueName: \"kubernetes.io/projected/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-kube-api-access-lk8lv\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:35:56.741315 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.741309    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-tls-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:35:56.741315 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:56.741318    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8-dshm\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:35:57.377649 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.377610    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r" podUID="56f6471d-0cdc-4796-92f9-33e223906222" containerName="main" containerID="cri-o://315a9bbd3f109d25683002c5d04a31ab491ae261742a6a97c74812bc896fcc9c" gracePeriod=45
Sep 18 16:35:57.506177 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.506146    2554 generic.go:356] "Generic (PLEG): container finished" podID="56f6471d-0cdc-4796-92f9-33e223906222" containerID="315a9bbd3f109d25683002c5d04a31ab491ae261742a6a97c74812bc896fcc9c" exitCode=0
Sep 18 16:35:57.506318 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.506221    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r" event={"ID":"56f6471d-0cdc-4796-92f9-33e223906222","Type":"ContainerDied","Data":"315a9bbd3f109d25683002c5d04a31ab491ae261742a6a97c74812bc896fcc9c"}
Sep 18 16:35:57.507719 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.507673    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"
Sep 18 16:35:57.507858 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.507720    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466" event={"ID":"fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8","Type":"ContainerDied","Data":"98e5b01cf37805c5e606c0276da076308003988e25f69232aff11a644293a139"}
Sep 18 16:35:57.507858 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.507772    2554 scope.go:117] "RemoveContainer" containerID="6fcb776b48605aa1182226f1b4f4e8607567176639396824dfe6e3110388249a"
Sep 18 16:35:57.536506 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.536437    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"]
Sep 18 16:35:57.545373 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.545349    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a75117/auth-test-model-a75117-kserve-946674bdc-l2466"]
Sep 18 16:35:57.633583 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.633561    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:35:57.751798 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.751768    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9xwd\" (UniqueName: \"kubernetes.io/projected/56f6471d-0cdc-4796-92f9-33e223906222-kube-api-access-w9xwd\") pod \"56f6471d-0cdc-4796-92f9-33e223906222\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") "
Sep 18 16:35:57.751979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.751843    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-tmp-dir\") pod \"56f6471d-0cdc-4796-92f9-33e223906222\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") "
Sep 18 16:35:57.751979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.751879    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-home\") pod \"56f6471d-0cdc-4796-92f9-33e223906222\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") "
Sep 18 16:35:57.751979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.751904    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f6471d-0cdc-4796-92f9-33e223906222-tls-certs\") pod \"56f6471d-0cdc-4796-92f9-33e223906222\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") "
Sep 18 16:35:57.751979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.751949    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-model-cache\") pod \"56f6471d-0cdc-4796-92f9-33e223906222\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") "
Sep 18 16:35:57.752192 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.751988    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-dshm\") pod \"56f6471d-0cdc-4796-92f9-33e223906222\" (UID: \"56f6471d-0cdc-4796-92f9-33e223906222\") "
Sep 18 16:35:57.752192 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.752117    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56f6471d-0cdc-4796-92f9-33e223906222" (UID: "56f6471d-0cdc-4796-92f9-33e223906222"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:35:57.752192 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.752134    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-home" (OuterVolumeSpecName: "home") pod "56f6471d-0cdc-4796-92f9-33e223906222" (UID: "56f6471d-0cdc-4796-92f9-33e223906222"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:35:57.752366 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.752216    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-model-cache" (OuterVolumeSpecName: "model-cache") pod "56f6471d-0cdc-4796-92f9-33e223906222" (UID: "56f6471d-0cdc-4796-92f9-33e223906222"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:35:57.752366 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.752303    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-model-cache\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:35:57.752366 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.752316    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-tmp-dir\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:35:57.752366 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.752326    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-home\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:35:57.754221 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.754187    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f6471d-0cdc-4796-92f9-33e223906222-kube-api-access-w9xwd" (OuterVolumeSpecName: "kube-api-access-w9xwd") pod "56f6471d-0cdc-4796-92f9-33e223906222" (UID: "56f6471d-0cdc-4796-92f9-33e223906222"). InnerVolumeSpecName "kube-api-access-w9xwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:35:57.754221 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.754206    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-dshm" (OuterVolumeSpecName: "dshm") pod "56f6471d-0cdc-4796-92f9-33e223906222" (UID: "56f6471d-0cdc-4796-92f9-33e223906222"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:35:57.754221 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.754215    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56f6471d-0cdc-4796-92f9-33e223906222-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56f6471d-0cdc-4796-92f9-33e223906222" (UID: "56f6471d-0cdc-4796-92f9-33e223906222"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:35:57.853315 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.853254    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f6471d-0cdc-4796-92f9-33e223906222-tls-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:35:57.853315 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.853275    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f6471d-0cdc-4796-92f9-33e223906222-dshm\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:35:57.853315 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:57.853283    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9xwd\" (UniqueName: \"kubernetes.io/projected/56f6471d-0cdc-4796-92f9-33e223906222-kube-api-access-w9xwd\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:35:58.513660 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:58.513624    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r" event={"ID":"56f6471d-0cdc-4796-92f9-33e223906222","Type":"ContainerDied","Data":"f4df13efb4971228934bb331034049ae16d0f160ccd91acdd7dc9c527b04a7bc"}
Sep 18 16:35:58.514208 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:58.513632    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"
Sep 18 16:35:58.514208 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:58.513672    2554 scope.go:117] "RemoveContainer" containerID="315a9bbd3f109d25683002c5d04a31ab491ae261742a6a97c74812bc896fcc9c"
Sep 18 16:35:58.540294 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:58.540272    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"]
Sep 18 16:35:58.554410 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:58.554382    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-11725e/auth-test-model-11725e-kserve-67c56d9947-5zg7r"]
Sep 18 16:35:59.277692 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:59.277661    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56f6471d-0cdc-4796-92f9-33e223906222" path="/var/lib/kubelet/pods/56f6471d-0cdc-4796-92f9-33e223906222/volumes"
Sep 18 16:35:59.278062 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:35:59.278050    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8" path="/var/lib/kubelet/pods/fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8/volumes"
Sep 18 16:36:07.490713 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:36:07.490685    2554 scope.go:117] "RemoveContainer" containerID="90093d39e56536b53b081e98e0c3416a1e3f1d7bfb792a45369b78455ed237e5"
Sep 18 16:38:03.133138 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:38:03.133105    2554 generic.go:356] "Generic (PLEG): container finished" podID="ad3586f9-9035-40c1-bf26-143eeb8ff319" containerID="8a197b6472763abacc1c82523b1a45616f445b38623fc8ae8fbb86a61f162311" exitCode=1
Sep 18 16:38:03.133575 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:38:03.133177    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8" event={"ID":"ad3586f9-9035-40c1-bf26-143eeb8ff319","Type":"ContainerDied","Data":"8a197b6472763abacc1c82523b1a45616f445b38623fc8ae8fbb86a61f162311"}
Sep 18 16:38:03.133575 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:38:03.133460    2554 scope.go:117] "RemoveContainer" containerID="8a197b6472763abacc1c82523b1a45616f445b38623fc8ae8fbb86a61f162311"
Sep 18 16:38:03.137378 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:38:03.137360    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 18 16:38:03.347338 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:38:03.347306    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:38:04.140133 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:38:04.140098    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8" event={"ID":"ad3586f9-9035-40c1-bf26-143eeb8ff319","Type":"ContainerStarted","Data":"3e7b2761f1dc419ece89a11e5b510a7994d757072c4ace66ebe69d72c12f0783"}
Sep 18 16:38:04.140549 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:38:04.140157    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:38:15.146647 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:38:15.146612    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:38:19.917381 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:38:19.917326    2554 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:38:26.979651 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:38:26.979593    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:38:16Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:38:16Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:38:16Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:38:16Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-130-195.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:38:36.980000 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:38:36.979957    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:38:37.138287 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:38:37.138179    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{dns-operator-controller-manager-7b84cd7c9c-vrmv8.18d677ac2327cee4  kuadrant-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kuadrant-system,Name:dns-operator-controller-manager-7b84cd7c9c-vrmv8,UID:ad3586f9-9035-40c1-bf26-143eeb8ff319,APIVersion:v1,ResourceVersion:16928,FieldPath:spec.containers{manager},},Reason:Pulled,Message:Container image \"registry.redhat.io/rhcl-1/dns-rhel9-operator@sha256:d82688172d312a44d9978eacf7bd9e0402e9ba2967daf70b2ef30707f41faf4a\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:38:03.134242532 +0000 UTC m=+1978.081741283,LastTimestamp:2026-09-18 16:38:03.134242532 +0000 UTC m=+1978.081741283,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:38:46.087160 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:38:46.087107    2554 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:38:46.980644 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:38:46.980601    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:38:56.087546 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:38:56.087488    2554 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:38:56.981200 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:38:56.981163    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:39:03.136018 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:39:03.135982    2554 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods dns-operator-controller-manager-7b84cd7c9c-vrmv8)" podUID="ad3586f9-9035-40c1-bf26-143eeb8ff319" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:39:06.087780 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:39:06.087735    2554 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:39:06.982176 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:39:06.982138    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:39:06.982176 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:39:06.982167    2554 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Sep 18 16:39:11.141839 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:39:11.141748    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{dns-operator-controller-manager-7b84cd7c9c-vrmv8.18d676533246cd10  kuadrant-system   17334 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kuadrant-system,Name:dns-operator-controller-manager-7b84cd7c9c-vrmv8,UID:ad3586f9-9035-40c1-bf26-143eeb8ff319,APIVersion:v1,ResourceVersion:16928,FieldPath:spec.containers{manager},},Reason:Created,Message:Created container: manager,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:13:21 +0000 UTC,LastTimestamp:2026-09-18 16:38:03.195710897 +0000 UTC m=+1978.143209662,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:39:16.088284 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:39:16.088234    2554 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:39:26.089026 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:39:26.088934    2554 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:39:26.089026 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:39:26.088978    2554 controller.go:115] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease"
Sep 18 16:39:27.265424 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:39:27.265383    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:39:17Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:39:17Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:39:17Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:39:17Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-130-195.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal/status?timeout=10s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:39:36.090053 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:39:36.090005    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="200ms"
Sep 18 16:39:37.266360 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:39:37.266324    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:39:45.145722 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:39:45.145629    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{dns-operator-controller-manager-7b84cd7c9c-vrmv8.18d6765332a8ef5f  kuadrant-system   17335 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kuadrant-system,Name:dns-operator-controller-manager-7b84cd7c9c-vrmv8,UID:ad3586f9-9035-40c1-bf26-143eeb8ff319,APIVersion:v1,ResourceVersion:16928,FieldPath:spec.containers{manager},},Reason:Started,Message:Started container manager,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:13:21 +0000 UTC,LastTimestamp:2026-09-18 16:38:03.203452621 +0000 UTC m=+1978.150951386,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:39:46.291709 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:39:46.291668    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded" interval="400ms"
Sep 18 16:39:47.266765 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:39:47.266729    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:39:49.646491 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:39:49.646461    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4bh4k_db1edd1b-c2ec-4a6a-b365-9305e87f8471/volume-data-source-validator/0.log"
Sep 18 16:39:49.646902 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:39:49.646510    2554 generic.go:356] "Generic (PLEG): container finished" podID="db1edd1b-c2ec-4a6a-b365-9305e87f8471" containerID="5e6aec84b5039c6113276e585d3fd07eadcdfe31f09a41dd4969ba3218c0bc46" exitCode=1
Sep 18 16:39:49.646902 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:39:49.646595    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k" event={"ID":"db1edd1b-c2ec-4a6a-b365-9305e87f8471","Type":"ContainerDied","Data":"5e6aec84b5039c6113276e585d3fd07eadcdfe31f09a41dd4969ba3218c0bc46"}
Sep 18 16:39:49.646980 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:39:49.646917    2554 scope.go:117] "RemoveContainer" containerID="5e6aec84b5039c6113276e585d3fd07eadcdfe31f09a41dd4969ba3218c0bc46"
Sep 18 16:39:50.653193 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:39:50.653167    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4bh4k_db1edd1b-c2ec-4a6a-b365-9305e87f8471/volume-data-source-validator/0.log"
Sep 18 16:39:50.653585 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:39:50.653277    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k" event={"ID":"db1edd1b-c2ec-4a6a-b365-9305e87f8471","Type":"ContainerStarted","Data":"3d6f76e3523637046ad700de6552ccd3290cb5f10706b5fca29e8dfe746c301a"}
Sep 18 16:39:56.693477 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:39:56.693434    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="800ms"
Sep 18 16:39:57.267088 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:39:57.267047    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:40:03.140851 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:40:03.140813    2554 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods dns-operator-controller-manager-7b84cd7c9c-vrmv8)" podUID="ad3586f9-9035-40c1-bf26-143eeb8ff319" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:40:07.268117 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:40:07.268074    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:40:07.268117 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:40:07.268106    2554 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Sep 18 16:40:07.345706 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:40:07.345677    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4bh4k_db1edd1b-c2ec-4a6a-b365-9305e87f8471/volume-data-source-validator/0.log"
Sep 18 16:40:07.359393 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:40:07.359373    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4bh4k_db1edd1b-c2ec-4a6a-b365-9305e87f8471/volume-data-source-validator/0.log"
Sep 18 16:40:07.494632 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:40:07.494596    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded" interval="1.6s"
Sep 18 16:40:19.096059 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:40:19.096007    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded" interval="3.2s"
Sep 18 16:40:23.651066 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:40:23.650980    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{volume-data-source-validator-5c4db8bdf9-4bh4k.18d677c4efd99c5a  openshift-cluster-storage-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-cluster-storage-operator,Name:volume-data-source-validator-5c4db8bdf9-4bh4k,UID:db1edd1b-c2ec-4a6a-b365-9305e87f8471,APIVersion:v1,ResourceVersion:7149,FieldPath:spec.containers{volume-data-source-validator},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcf92586b0800da0c7611d579e19b109bc966fa0cbc624de06c4a2b416d8cce7\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:39:49.64766217 +0000 UTC m=+2084.595160922,LastTimestamp:2026-09-18 16:39:49.64766217 +0000 UTC m=+2084.595160922,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:40:27.332193 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:40:27.332149    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:40:17Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:40:17Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:40:17Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:40:17Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-130-195.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:40:32.297347 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:40:32.297311    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="6.4s"
Sep 18 16:40:37.333052 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:40:37.333010    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:40:47.333992 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:40:47.333901    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:40:48.698798 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:40:48.698749    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:40:57.334133 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:40:57.334090    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:40:57.654403 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:40:57.654259    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{volume-data-source-validator-5c4db8bdf9-4bh4k.18d675f762493dd7  openshift-cluster-storage-operator   7297 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-cluster-storage-operator,Name:volume-data-source-validator-5c4db8bdf9-4bh4k,UID:db1edd1b-c2ec-4a6a-b365-9305e87f8471,APIVersion:v1,ResourceVersion:7149,FieldPath:spec.containers{volume-data-source-validator},},Reason:Created,Message:Created container: volume-data-source-validator,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:06:47 +0000 UTC,LastTimestamp:2026-09-18 16:39:49.694231665 +0000 UTC m=+2084.641730441,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:41:05.700628 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:41:05.700577    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:41:07.334668 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:41:07.334628    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:41:07.334668 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:41:07.334657    2554 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Sep 18 16:41:14.060285 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:41:14.060253    2554 generic.go:356] "Generic (PLEG): container finished" podID="8a5a8e40-2a6b-4289-be46-bf1e50af868e" containerID="1720953806fefc41b2941d2b1433a47d5d6d88ee767d6cb388c5183333b10358" exitCode=0
Sep 18 16:41:14.060737 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:41:14.060313    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp" event={"ID":"8a5a8e40-2a6b-4289-be46-bf1e50af868e","Type":"ContainerDied","Data":"1720953806fefc41b2941d2b1433a47d5d6d88ee767d6cb388c5183333b10358"}
Sep 18 16:41:14.060737 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:41:14.060354    2554 scope.go:117] "RemoveContainer" containerID="7d4d7a26450a09481d5170ae2708f66657cc9d83bb0cb0c792a89d1468cd61db"
Sep 18 16:41:14.060737 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:41:14.060672    2554 scope.go:117] "RemoveContainer" containerID="1720953806fefc41b2941d2b1433a47d5d6d88ee767d6cb388c5183333b10358"
Sep 18 16:41:15.067278 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:41:15.067242    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp" event={"ID":"8a5a8e40-2a6b-4289-be46-bf1e50af868e","Type":"ContainerStarted","Data":"e6c0b7ec52d043229cd8efcba67eca8b709ede4c2c1136272ce13296eaae17f7"}
Sep 18 16:41:16.706163 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:41:16.706094    2554 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"db1edd1b-c2ec-4a6a-b365-9305e87f8471\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-09-18T16:39:50Z\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-09-18T16:39:50Z\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"50Mi\\\"},\\\"containerID\\\":\\\"cri-o://3d6f76e3523637046ad700de6552ccd3290cb5f10706b5fca29e8dfe746c301a\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcf92586b0800da0c7611d579e19b109bc966fa0cbc624de06c4a2b416d8cce7\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c35aaec06e2a5e18026e96812471952fe5653f94e5ec79bc15f360248c9b2f80\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://5e6aec84b5039c6113276e585d3fd07eadcdfe31f09a41dd4969ba3218c0bc46\\\",\\\"exitCode\\\":1,\\\"finishedAt\\\":\\\"2026-09-18T16:39:48Z\\\",\\\"message\\\":\\\"ing to acquire leader lease openshift-cluster-storage-operator/data-source-validator-leader...\\\\nI0918 16:06:47.336635       1 leaderelection.go:271] successfully acquired lease openshift-cluster-storage-operator/data-source-validator-leader\\\\nI0918 16:06:47.336800       1 leader_election.go:184] \\\\\\\"became leader, starting\\\\\\\"\\\\nI0918 16:06:47.337023       1 data_source_validator.go:107] Starting volume-data-source-validator controller\\\\nI0918 16:06:47.339394       1 reflector.go:436] \\\\\\\"Caches populated\\\\\\\" type=\\\\\\\"*v1.PersistentVolumeClaim\\\\\\\" reflector=\\\\\\\"k8s.io/client-go/informers/factory.go:160\\\\\\\"\\\\nI0918 16:06:47.339674       1 reflector.go:436] \\\\\\\"Caches populated\\\\\\\" type=\\\\\\\"populator.storage.k8s.io/v1beta1, Resource=volumepopulators\\\\\\\" reflector=\\\\\\\"k8s.io/client-go/dynamic/dynamicinformer/informer.go:108\\\\\\\"\\\\nE0918 16:38:35.742682       1 leaderelection.go:441] Failed to update lock optimistically: Timeout: request did not complete within requested timeout - context deadline exceeded, falling back to slow path\\\\nE0918 16:39:35.744938       1 leaderelection.go:448] error retrieving resource lock openshift-cluster-storage-operator/data-source-validator-leader: the server was unable to return a response in the time allotted, but may still be processing the request (get leases.coordination.k8s.io data-source-validator-leader)\\\\nE0918 16:39:48.740082       1 leaderelection.go:441] Failed to update lock optimistically: Put \\\\\\\"https://172.31.0.1:443/apis/coordination.k8s.io/v1/namespaces/openshift-cluster-storage-operator/leases/data-source-validator-leader\\\\\\\": context deadline exceeded, falling back to slow path\\\\nE0918 16:39:48.740152       1 leaderelection.go:448] error retrieving resource lock openshift-cluster-storage-operator/data-source-validator-leader: client rate limiter Wait returned an error: context deadline exceeded\\\\nI0918 16:39:48.740166       1 leaderelection.go:297] failed to renew lease openshift-cluster-storage-operator/data-source-validator-leader: context deadline exceeded\\\\nE0918 16:39:48.740209       1 leader_election.go:188] \\\\\\\"Stopped leading\\\\\\\"\\\\n\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-09-18T16:06:47Z\\\"}},\\\"name\\\":\\\"volume-data-source-validator\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"50Mi\\\"}},\\\"restartCount\\\":1,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-09-18T16:39:49Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000160000],\\\"uid\\\":1000160000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wp74t\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-cluster-storage-operator\"/\"volume-data-source-validator-5c4db8bdf9-4bh4k\": Timeout: request did not complete within requested timeout - context deadline exceeded" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k"
Sep 18 16:41:22.700926 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:41:22.700886    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:41:27.551957 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:41:27.551913    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:41:17Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:41:17Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:41:17Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:41:17Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-130-195.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal/status?timeout=10s\": context deadline exceeded"
Sep 18 16:41:31.657870 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:41:31.657781    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{volume-data-source-validator-5c4db8bdf9-4bh4k.18d675f762b65afe  openshift-cluster-storage-operator   7298 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-cluster-storage-operator,Name:volume-data-source-validator-5c4db8bdf9-4bh4k,UID:db1edd1b-c2ec-4a6a-b365-9305e87f8471,APIVersion:v1,ResourceVersion:7149,FieldPath:spec.containers{volume-data-source-validator},},Reason:Started,Message:Started container volume-data-source-validator,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:06:47 +0000 UTC,LastTimestamp:2026-09-18 16:39:49.702138573 +0000 UTC m=+2084.649637346,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:41:39.702579 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:41:39.702461    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:41:40.192698 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:41:40.192664    2554 generic.go:356] "Generic (PLEG): container finished" podID="720642cf-540a-42d3-ab46-487cf5a95e72" containerID="9921139325e7f8035cb3368ff0a010aac348cb93f1f6a9148ad976c4643013e9" exitCode=1
Sep 18 16:41:40.192879 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:41:40.192724    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6" event={"ID":"720642cf-540a-42d3-ab46-487cf5a95e72","Type":"ContainerDied","Data":"9921139325e7f8035cb3368ff0a010aac348cb93f1f6a9148ad976c4643013e9"}
Sep 18 16:41:40.193055 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:41:40.193042    2554 scope.go:117] "RemoveContainer" containerID="9921139325e7f8035cb3368ff0a010aac348cb93f1f6a9148ad976c4643013e9"
Sep 18 16:41:41.200122 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:41:41.200089    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6" event={"ID":"720642cf-540a-42d3-ab46-487cf5a95e72","Type":"ContainerStarted","Data":"187da15814c61dbdb246241efdf6d9ab5cf28793acf2eeedb848d100d7365859"}
Sep 18 16:41:41.200515 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:41:41.200292    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:41:52.207039 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:41:52.207006    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:41:56.703692 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:41:56.703653    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:42:05.661731 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:42:05.661644    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{kube-storage-version-migrator-operator-55889f7b47-44mkp.18d6760aa88090fa  openshift-kube-storage-version-migrator-operator   9643 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-kube-storage-version-migrator-operator,Name:kube-storage-version-migrator-operator-55889f7b47-44mkp,UID:8a5a8e40-2a6b-4289-be46-bf1e50af868e,APIVersion:v1,ResourceVersion:7197,FieldPath:spec.containers{kube-storage-version-migrator-operator},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19f97ecb88e6cbef776b9ec3c20fde1b9bf3e3584afdd21ed2dab9d4b0541102\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:08:10 +0000 UTC,LastTimestamp:2026-09-18 16:41:14.061567749 +0000 UTC m=+2169.009066506,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:42:13.705179 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:42:13.705118    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:42:17.376982 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:42:17.376896    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:42:07Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:42:07Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:42:07Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:42:07Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-130-195.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:42:27.377808 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:42:27.377770    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:42:30.705674 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:42:30.705637    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded" interval="7s"
Sep 18 16:42:37.378703 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:42:37.378646    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:42:39.667874 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:42:39.667773    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{kube-storage-version-migrator-operator-55889f7b47-44mkp.18d675f7e25cd594  openshift-kube-storage-version-migrator-operator   9663 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-kube-storage-version-migrator-operator,Name:kube-storage-version-migrator-operator-55889f7b47-44mkp,UID:8a5a8e40-2a6b-4289-be46-bf1e50af868e,APIVersion:v1,ResourceVersion:7197,FieldPath:spec.containers{kube-storage-version-migrator-operator},},Reason:Created,Message:Created container: kube-storage-version-migrator-operator,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:06:49 +0000 UTC,LastTimestamp:2026-09-18 16:41:14.11268722 +0000 UTC m=+2169.060185985,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:42:47.379764 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:42:47.379726    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:42:47.707056 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:42:47.707008    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:42:57.380454 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:42:57.380419    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:42:57.380454 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:42:57.380446    2554 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Sep 18 16:43:00.576163 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:43:00.576138    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-44mkp_8a5a8e40-2a6b-4289-be46-bf1e50af868e/kube-storage-version-migrator-operator/2.log"
Sep 18 16:43:00.576622 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:43:00.576500    2554 generic.go:356] "Generic (PLEG): container finished" podID="8a5a8e40-2a6b-4289-be46-bf1e50af868e" containerID="e6c0b7ec52d043229cd8efcba67eca8b709ede4c2c1136272ce13296eaae17f7" exitCode=255
Sep 18 16:43:00.576622 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:43:00.576579    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp" event={"ID":"8a5a8e40-2a6b-4289-be46-bf1e50af868e","Type":"ContainerDied","Data":"e6c0b7ec52d043229cd8efcba67eca8b709ede4c2c1136272ce13296eaae17f7"}
Sep 18 16:43:00.576710 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:43:00.576638    2554 scope.go:117] "RemoveContainer" containerID="1720953806fefc41b2941d2b1433a47d5d6d88ee767d6cb388c5183333b10358"
Sep 18 16:43:00.577033 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:43:00.577017    2554 scope.go:117] "RemoveContainer" containerID="e6c0b7ec52d043229cd8efcba67eca8b709ede4c2c1136272ce13296eaae17f7"
Sep 18 16:43:00.577247 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:43:00.577228    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-storage-version-migrator-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=kube-storage-version-migrator-operator pod=kube-storage-version-migrator-operator-55889f7b47-44mkp_openshift-kube-storage-version-migrator-operator(8a5a8e40-2a6b-4289-be46-bf1e50af868e)\"" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp" podUID="8a5a8e40-2a6b-4289-be46-bf1e50af868e"
Sep 18 16:43:01.582833 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:43:01.582806    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-44mkp_8a5a8e40-2a6b-4289-be46-bf1e50af868e/kube-storage-version-migrator-operator/2.log"
Sep 18 16:43:04.708014 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:43:04.707970    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:43:13.271383 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:43:13.271355    2554 scope.go:117] "RemoveContainer" containerID="e6c0b7ec52d043229cd8efcba67eca8b709ede4c2c1136272ce13296eaae17f7"
Sep 18 16:43:13.644542 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:43:13.644443    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-44mkp_8a5a8e40-2a6b-4289-be46-bf1e50af868e/kube-storage-version-migrator-operator/2.log"
Sep 18 16:43:13.644542 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:43:13.644488    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-44mkp" event={"ID":"8a5a8e40-2a6b-4289-be46-bf1e50af868e","Type":"ContainerStarted","Data":"093684fee68451891171e303f17abb122676ddcfbbe9f444dc5c90331e10db7f"}
Sep 18 16:43:13.671272 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:43:13.671197    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{kube-storage-version-migrator-operator-55889f7b47-44mkp.18d675f7e2fa990c  openshift-kube-storage-version-migrator-operator   9666 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-kube-storage-version-migrator-operator,Name:kube-storage-version-migrator-operator-55889f7b47-44mkp,UID:8a5a8e40-2a6b-4289-be46-bf1e50af868e,APIVersion:v1,ResourceVersion:7197,FieldPath:spec.containers{kube-storage-version-migrator-operator},},Reason:Started,Message:Started container kube-storage-version-migrator-operator,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:06:49 +0000 UTC,LastTimestamp:2026-09-18 16:41:14.120033515 +0000 UTC m=+2169.067532279,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:43:17.419735 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:43:17.419691    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:43:07Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:43:07Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:43:07Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:43:07Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-130-195.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal/status?timeout=10s\": context deadline exceeded"
Sep 18 16:43:21.709598 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:43:21.709557    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded" interval="7s"
Sep 18 16:43:27.420051 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:43:27.420015    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:43:37.421202 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:43:37.421169    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:43:38.710575 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:43:38.710510    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:43:47.675068 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:43:47.674930    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{lws-controller-manager-6847d847f7-2hft6.18d677deace9896e  openshift-lws-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-lws-operator,Name:lws-controller-manager-6847d847f7-2hft6,UID:720642cf-540a-42d3-ab46-487cf5a95e72,APIVersion:v1,ResourceVersion:13570,FieldPath:spec.containers{manager},},Reason:Pulled,Message:Container image \"registry.redhat.io/leader-worker-set/lws-rhel9@sha256:affb303b1173c273231bb50fef07310b0e220d2f08bfc0aa5912d0825e3e0d4f\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:41:40.193782126 +0000 UTC m=+2195.141280879,LastTimestamp:2026-09-18 16:41:40.193782126 +0000 UTC m=+2195.141280879,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:43:49.914626 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:43:49.914587    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:43:39Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:43:39Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:43:39Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:43:39Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-130-195.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:44:14.946574 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:14.946515    2554 generic.go:356] "Generic (PLEG): container finished" podID="ad3586f9-9035-40c1-bf26-143eeb8ff319" containerID="3e7b2761f1dc419ece89a11e5b510a7994d757072c4ace66ebe69d72c12f0783" exitCode=1
Sep 18 16:44:14.947079 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:14.946595    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8" event={"ID":"ad3586f9-9035-40c1-bf26-143eeb8ff319","Type":"ContainerDied","Data":"3e7b2761f1dc419ece89a11e5b510a7994d757072c4ace66ebe69d72c12f0783"}
Sep 18 16:44:14.947079 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:14.946670    2554 scope.go:117] "RemoveContainer" containerID="8a197b6472763abacc1c82523b1a45616f445b38623fc8ae8fbb86a61f162311"
Sep 18 16:44:14.947079 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:14.947037    2554 scope.go:117] "RemoveContainer" containerID="3e7b2761f1dc419ece89a11e5b510a7994d757072c4ace66ebe69d72c12f0783"
Sep 18 16:44:14.947374 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:44:14.947343    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=dns-operator-controller-manager-7b84cd7c9c-vrmv8_kuadrant-system(ad3586f9-9035-40c1-bf26-143eeb8ff319)\"" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8" podUID="ad3586f9-9035-40c1-bf26-143eeb8ff319"
Sep 18 16:44:14.948764 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:14.948745    2554 generic.go:356] "Generic (PLEG): container finished" podID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerID="b1ac7815ed22d38535e520a06fdad108186adfd4f8ad23fd553abb4e925c4e77" exitCode=1
Sep 18 16:44:14.948848 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:14.948820    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" event={"ID":"64d4aca4-7571-4c30-a7c8-b1747b3635b6","Type":"ContainerDied","Data":"b1ac7815ed22d38535e520a06fdad108186adfd4f8ad23fd553abb4e925c4e77"}
Sep 18 16:44:14.949139 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:14.949126    2554 scope.go:117] "RemoveContainer" containerID="b1ac7815ed22d38535e520a06fdad108186adfd4f8ad23fd553abb4e925c4e77"
Sep 18 16:44:14.949392 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:14.949382    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:44:15.145681 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:15.145591    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:44:15.956300 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:15.956267    2554 scope.go:117] "RemoveContainer" containerID="3e7b2761f1dc419ece89a11e5b510a7994d757072c4ace66ebe69d72c12f0783"
Sep 18 16:44:15.956777 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:44:15.956487    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=dns-operator-controller-manager-7b84cd7c9c-vrmv8_kuadrant-system(ad3586f9-9035-40c1-bf26-143eeb8ff319)\"" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8" podUID="ad3586f9-9035-40c1-bf26-143eeb8ff319"
Sep 18 16:44:15.958258 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:15.958237    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" event={"ID":"64d4aca4-7571-4c30-a7c8-b1747b3635b6","Type":"ContainerStarted","Data":"0210f35f117e793f7c4d771219e3600f56bd2b8a1d9a795b2fe673f172187152"}
Sep 18 16:44:15.958448 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:15.958433    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:44:18.740061 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:18.740025    2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:44:18.740612 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:18.740509    2554 scope.go:117] "RemoveContainer" containerID="3e7b2761f1dc419ece89a11e5b510a7994d757072c4ace66ebe69d72c12f0783"
Sep 18 16:44:18.740787 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:44:18.740762    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=dns-operator-controller-manager-7b84cd7c9c-vrmv8_kuadrant-system(ad3586f9-9035-40c1-bf26-143eeb8ff319)\"" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8" podUID="ad3586f9-9035-40c1-bf26-143eeb8ff319"
Sep 18 16:44:20.890431 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:44:20.890381    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:44:10Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:44:10Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:44:10Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:44:10Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-130-195.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:44:21.202661 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:44:21.202617    2554 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:44:21.678359 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:44:21.678271    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{lws-controller-manager-6847d847f7-2hft6.18d6763c873c14a1  openshift-lws-operator   14022 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-lws-operator,Name:lws-controller-manager-6847d847f7-2hft6,UID:720642cf-540a-42d3-ab46-487cf5a95e72,APIVersion:v1,ResourceVersion:13570,FieldPath:spec.containers{manager},},Reason:Created,Message:Created container: manager,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:11:44 +0000 UTC,LastTimestamp:2026-09-18 16:41:40.248258851 +0000 UTC m=+2195.195757602,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:44:26.965807 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:26.965766    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:44:30.272070 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:30.272037    2554 scope.go:117] "RemoveContainer" containerID="3e7b2761f1dc419ece89a11e5b510a7994d757072c4ace66ebe69d72c12f0783"
Sep 18 16:44:30.276289 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:30.276268    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 18 16:44:30.891373 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:44:30.891334    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:44:31.033546 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:31.033502    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8" event={"ID":"ad3586f9-9035-40c1-bf26-143eeb8ff319","Type":"ContainerStarted","Data":"e61fafaffdcdb31cbd166dbf14e207c7a48768e1139df74026bdb54b429368f8"}
Sep 18 16:44:31.033742 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:31.033725    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:44:31.203636 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:44:31.203592    2554 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:44:33.014515 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:44:33.014483    2554 controller.go:195] "Failed to update lease" err="Operation cannot be fulfilled on leases.coordination.k8s.io \"ip-10-0-130-195.ec2.internal\": the object has been modified; please apply your changes to the latest version and try again"
Sep 18 16:44:36.965239 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:36.965196    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:44:42.041225 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:42.041193    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:44:43.922887 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:43.922850    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:44:46.964548 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:46.964492    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:44:55.682120 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:44:55.682032    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{lws-controller-manager-6847d847f7-2hft6.18d6763c87a7b4ef  openshift-lws-operator   14023 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-lws-operator,Name:lws-controller-manager-6847d847f7-2hft6,UID:720642cf-540a-42d3-ab46-487cf5a95e72,APIVersion:v1,ResourceVersion:13570,FieldPath:spec.containers{manager},},Reason:Started,Message:Started container manager,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:11:44 +0000 UTC,LastTimestamp:2026-09-18 16:41:40.256201298 +0000 UTC m=+2195.203700069,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:44:56.964410 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:44:56.964370    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:45:03.923368 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:03.923320    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:45:06.965162 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:06.965120    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:45:07.218467 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:07.218389    2554 generic.go:356] "Generic (PLEG): container finished" podID="720642cf-540a-42d3-ab46-487cf5a95e72" containerID="187da15814c61dbdb246241efdf6d9ab5cf28793acf2eeedb848d100d7365859" exitCode=1
Sep 18 16:45:07.218467 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:07.218456    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6" event={"ID":"720642cf-540a-42d3-ab46-487cf5a95e72","Type":"ContainerDied","Data":"187da15814c61dbdb246241efdf6d9ab5cf28793acf2eeedb848d100d7365859"}
Sep 18 16:45:07.218676 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:07.218512    2554 scope.go:117] "RemoveContainer" containerID="9921139325e7f8035cb3368ff0a010aac348cb93f1f6a9148ad976c4643013e9"
Sep 18 16:45:07.218857 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:07.218842    2554 scope.go:117] "RemoveContainer" containerID="187da15814c61dbdb246241efdf6d9ab5cf28793acf2eeedb848d100d7365859"
Sep 18 16:45:07.219086 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:45:07.219066    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=lws-controller-manager-6847d847f7-2hft6_openshift-lws-operator(720642cf-540a-42d3-ab46-487cf5a95e72)\"" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6" podUID="720642cf-540a-42d3-ab46-487cf5a95e72"
Sep 18 16:45:07.376124 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:07.376097    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-44mkp_8a5a8e40-2a6b-4289-be46-bf1e50af868e/kube-storage-version-migrator-operator/2.log"
Sep 18 16:45:07.376773 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:07.376757    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4bh4k_db1edd1b-c2ec-4a6a-b365-9305e87f8471/volume-data-source-validator/0.log"
Sep 18 16:45:07.390884 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:07.390865    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-44mkp_8a5a8e40-2a6b-4289-be46-bf1e50af868e/kube-storage-version-migrator-operator/2.log"
Sep 18 16:45:07.391490 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:07.391473    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4bh4k_db1edd1b-c2ec-4a6a-b365-9305e87f8471/volume-data-source-validator/0.log"
Sep 18 16:45:08.225709 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:08.225673    2554 generic.go:356] "Generic (PLEG): container finished" podID="ad3586f9-9035-40c1-bf26-143eeb8ff319" containerID="e61fafaffdcdb31cbd166dbf14e207c7a48768e1139df74026bdb54b429368f8" exitCode=1
Sep 18 16:45:08.226159 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:08.225745    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8" event={"ID":"ad3586f9-9035-40c1-bf26-143eeb8ff319","Type":"ContainerDied","Data":"e61fafaffdcdb31cbd166dbf14e207c7a48768e1139df74026bdb54b429368f8"}
Sep 18 16:45:08.226159 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:08.225810    2554 scope.go:117] "RemoveContainer" containerID="3e7b2761f1dc419ece89a11e5b510a7994d757072c4ace66ebe69d72c12f0783"
Sep 18 16:45:08.226159 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:08.226144    2554 scope.go:117] "RemoveContainer" containerID="e61fafaffdcdb31cbd166dbf14e207c7a48768e1139df74026bdb54b429368f8"
Sep 18 16:45:08.226418 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:45:08.226398    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=dns-operator-controller-manager-7b84cd7c9c-vrmv8_kuadrant-system(ad3586f9-9035-40c1-bf26-143eeb8ff319)\"" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8" podUID="ad3586f9-9035-40c1-bf26-143eeb8ff319"
Sep 18 16:45:12.038806 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:12.038764    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:45:12.039281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:12.039103    2554 scope.go:117] "RemoveContainer" containerID="e61fafaffdcdb31cbd166dbf14e207c7a48768e1139df74026bdb54b429368f8"
Sep 18 16:45:12.039281 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:45:12.039267    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=dns-operator-controller-manager-7b84cd7c9c-vrmv8_kuadrant-system(ad3586f9-9035-40c1-bf26-143eeb8ff319)\"" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8" podUID="ad3586f9-9035-40c1-bf26-143eeb8ff319"
Sep 18 16:45:12.205340 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:12.205307    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:45:12.205668 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:12.205654    2554 scope.go:117] "RemoveContainer" containerID="187da15814c61dbdb246241efdf6d9ab5cf28793acf2eeedb848d100d7365859"
Sep 18 16:45:12.205824 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:45:12.205807    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=lws-controller-manager-6847d847f7-2hft6_openshift-lws-operator(720642cf-540a-42d3-ab46-487cf5a95e72)\"" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6" podUID="720642cf-540a-42d3-ab46-487cf5a95e72"
Sep 18 16:45:13.559405 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:45:13.559349    2554 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:45:16.965155 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:16.965110    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:45:18.740383 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:18.740342    2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:45:18.740898 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:18.740726    2554 scope.go:117] "RemoveContainer" containerID="e61fafaffdcdb31cbd166dbf14e207c7a48768e1139df74026bdb54b429368f8"
Sep 18 16:45:18.740942 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:45:18.740897    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=dns-operator-controller-manager-7b84cd7c9c-vrmv8_kuadrant-system(ad3586f9-9035-40c1-bf26-143eeb8ff319)\"" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8" podUID="ad3586f9-9035-40c1-bf26-143eeb8ff319"
Sep 18 16:45:21.727548 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:21.727486    2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:45:21.728053 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:21.727992    2554 scope.go:117] "RemoveContainer" containerID="187da15814c61dbdb246241efdf6d9ab5cf28793acf2eeedb848d100d7365859"
Sep 18 16:45:22.301356 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:22.301322    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6" event={"ID":"720642cf-540a-42d3-ab46-487cf5a95e72","Type":"ContainerStarted","Data":"8726a31c89c1c79609e6e0fef12c2d3d0e45e99775fa8a67d87087085450ef83"}
Sep 18 16:45:22.301556 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:22.301539    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:45:23.559754 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:45:23.559710    2554 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:45:23.701096 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:45:23.701043    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:45:13Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:45:13Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:45:13Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:45:13Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-130-195.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal/status?timeout=10s\": context deadline exceeded"
Sep 18 16:45:23.923041 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:23.922950    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:45:23.923041 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:23.923024    2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:45:23.923564 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:23.923544    2554 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"0210f35f117e793f7c4d771219e3600f56bd2b8a1d9a795b2fe673f172187152"} pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" containerMessage="Container manager failed liveness probe, will be restarted"
Sep 18 16:45:23.923641 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:23.923590    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" containerID="cri-o://0210f35f117e793f7c4d771219e3600f56bd2b8a1d9a795b2fe673f172187152" gracePeriod=10
Sep 18 16:45:23.923705 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:23.923664    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:45:24.311584 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:24.311551    2554 generic.go:356] "Generic (PLEG): container finished" podID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerID="0210f35f117e793f7c4d771219e3600f56bd2b8a1d9a795b2fe673f172187152" exitCode=1
Sep 18 16:45:24.311744 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:24.311618    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" event={"ID":"64d4aca4-7571-4c30-a7c8-b1747b3635b6","Type":"ContainerDied","Data":"0210f35f117e793f7c4d771219e3600f56bd2b8a1d9a795b2fe673f172187152"}
Sep 18 16:45:24.311744 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:24.311679    2554 scope.go:117] "RemoveContainer" containerID="b1ac7815ed22d38535e520a06fdad108186adfd4f8ad23fd553abb4e925c4e77"
Sep 18 16:45:25.317933 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:25.317893    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" event={"ID":"64d4aca4-7571-4c30-a7c8-b1747b3635b6","Type":"ContainerStarted","Data":"a287cde99b1f9edd88637239a2e71895847839397c8e38a159f3c87456408f2d"}
Sep 18 16:45:25.318341 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:25.318033    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:45:29.271404 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:29.271376    2554 scope.go:117] "RemoveContainer" containerID="e61fafaffdcdb31cbd166dbf14e207c7a48768e1139df74026bdb54b429368f8"
Sep 18 16:45:29.339334 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:29.339307    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8" event={"ID":"ad3586f9-9035-40c1-bf26-143eeb8ff319","Type":"ContainerStarted","Data":"89b6dd468412a5f9e31038a537b6f2b83748e8a210476249288afb5ee532e1cf"}
Sep 18 16:45:29.339551 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:29.339517    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:45:29.687016 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:45:29.686927    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{kube-storage-version-migrator-operator-55889f7b47-44mkp.18d677f16422e6aa  openshift-kube-storage-version-migrator-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-kube-storage-version-migrator-operator,Name:kube-storage-version-migrator-operator-55889f7b47-44mkp,UID:8a5a8e40-2a6b-4289-be46-bf1e50af868e,APIVersion:v1,ResourceVersion:7197,FieldPath:spec.containers{kube-storage-version-migrator-operator},},Reason:BackOff,Message:Back-off restarting failed container kube-storage-version-migrator-operator in pod kube-storage-version-migrator-operator-55889f7b47-44mkp_openshift-kube-storage-version-migrator-operator(8a5a8e40-2a6b-4289-be46-bf1e50af868e),Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:43:00.577183402 +0000 UTC m=+2275.524682168,LastTimestamp:2026-09-18 16:43:00.577183402 +0000 UTC m=+2275.524682168,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:45:33.307948 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:33.307919    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:45:33.560381 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:45:33.560283    2554 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:45:33.701314 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:45:33.701277    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:45:36.324465 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:36.324429    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:45:40.346839 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:40.346801    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:45:43.560741 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:45:43.560694    2554 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:45:43.701655 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:45:43.701621    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:45:43.923297 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:43.923201    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:45:46.324044 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:46.324002    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:45:53.561781 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:45:53.561729    2554 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:45:53.561781 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:53.561780    2554 controller.go:115] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease"
Sep 18 16:45:53.702480 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:45:53.702435    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:45:56.323328 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:45:56.323287    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:46:03.562666 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:46:03.562623    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="200ms"
Sep 18 16:46:03.690904 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:46:03.690815    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{kube-storage-version-migrator-operator-55889f7b47-44mkp.18d6760aa88090fa  openshift-kube-storage-version-migrator-operator   9643 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-kube-storage-version-migrator-operator,Name:kube-storage-version-migrator-operator-55889f7b47-44mkp,UID:8a5a8e40-2a6b-4289-be46-bf1e50af868e,APIVersion:v1,ResourceVersion:7197,FieldPath:spec.containers{kube-storage-version-migrator-operator},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19f97ecb88e6cbef776b9ec3c20fde1b9bf3e3584afdd21ed2dab9d4b0541102\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:08:10 +0000 UTC,LastTimestamp:2026-09-18 16:43:13.272336578 +0000 UTC m=+2288.219835329,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:46:03.703461 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:46:03.703430    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:46:03.703461 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:46:03.703455    2554 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Sep 18 16:46:03.923689 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:03.923594    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:46:06.323744 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:06.323699    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:46:07.222008 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:46:07.221967    2554 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods lws-controller-manager-6847d847f7-2hft6)" podUID="720642cf-540a-42d3-ab46-487cf5a95e72" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:46:13.763443 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:46:13.763398    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="400ms"
Sep 18 16:46:16.323624 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:16.323579    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:46:23.922981 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:23.922931    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:46:23.923390 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:23.923018    2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:46:23.923514 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:23.923496    2554 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"a287cde99b1f9edd88637239a2e71895847839397c8e38a159f3c87456408f2d"} pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" containerMessage="Container manager failed liveness probe, will be restarted"
Sep 18 16:46:23.923594 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:23.923558    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" containerID="cri-o://a287cde99b1f9edd88637239a2e71895847839397c8e38a159f3c87456408f2d" gracePeriod=10
Sep 18 16:46:23.923664 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:23.923641    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:46:24.165034 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:46:24.164990    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded" interval="800ms"
Sep 18 16:46:24.608255 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:24.608153    2554 generic.go:356] "Generic (PLEG): container finished" podID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerID="a287cde99b1f9edd88637239a2e71895847839397c8e38a159f3c87456408f2d" exitCode=1
Sep 18 16:46:24.608255 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:24.608230    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" event={"ID":"64d4aca4-7571-4c30-a7c8-b1747b3635b6","Type":"ContainerDied","Data":"a287cde99b1f9edd88637239a2e71895847839397c8e38a159f3c87456408f2d"}
Sep 18 16:46:24.608622 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:24.608293    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" event={"ID":"64d4aca4-7571-4c30-a7c8-b1747b3635b6","Type":"ContainerStarted","Data":"4dfb7acccf5d6dacd2c38fdec8ccfd80ee7a71b764360398fbb908d71e5efc8b"}
Sep 18 16:46:24.608622 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:24.608320    2554 scope.go:117] "RemoveContainer" containerID="0210f35f117e793f7c4d771219e3600f56bd2b8a1d9a795b2fe673f172187152"
Sep 18 16:46:24.608622 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:24.608545    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:46:34.966339 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:46:34.966297    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded" interval="1.6s"
Sep 18 16:46:35.618191 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:35.618154    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:46:37.694893 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:46:37.694810    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{kube-storage-version-migrator-operator-55889f7b47-44mkp.18d675f7e25cd594  openshift-kube-storage-version-migrator-operator   9663 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-kube-storage-version-migrator-operator,Name:kube-storage-version-migrator-operator-55889f7b47-44mkp,UID:8a5a8e40-2a6b-4289-be46-bf1e50af868e,APIVersion:v1,ResourceVersion:7197,FieldPath:spec.containers{kube-storage-version-migrator-operator},},Reason:Created,Message:Created container: kube-storage-version-migrator-operator,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:06:49 +0000 UTC,LastTimestamp:2026-09-18 16:43:13.322556293 +0000 UTC m=+2288.270055058,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:46:43.922936 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:43.922896    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:46:45.616820 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:45.616781    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:46:46.568181 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:46:46.568135    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="3.2s"
Sep 18 16:46:46.719098 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:46.719070    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4bh4k_db1edd1b-c2ec-4a6a-b365-9305e87f8471/volume-data-source-validator/1.log"
Sep 18 16:46:46.719621 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:46.719603    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4bh4k_db1edd1b-c2ec-4a6a-b365-9305e87f8471/volume-data-source-validator/0.log"
Sep 18 16:46:46.719687 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:46.719638    2554 generic.go:356] "Generic (PLEG): container finished" podID="db1edd1b-c2ec-4a6a-b365-9305e87f8471" containerID="3d6f76e3523637046ad700de6552ccd3290cb5f10706b5fca29e8dfe746c301a" exitCode=1
Sep 18 16:46:46.719741 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:46.719715    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k" event={"ID":"db1edd1b-c2ec-4a6a-b365-9305e87f8471","Type":"ContainerDied","Data":"3d6f76e3523637046ad700de6552ccd3290cb5f10706b5fca29e8dfe746c301a"}
Sep 18 16:46:46.719792 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:46.719780    2554 scope.go:117] "RemoveContainer" containerID="5e6aec84b5039c6113276e585d3fd07eadcdfe31f09a41dd4969ba3218c0bc46"
Sep 18 16:46:46.720065 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:46.720052    2554 scope.go:117] "RemoveContainer" containerID="3d6f76e3523637046ad700de6552ccd3290cb5f10706b5fca29e8dfe746c301a"
Sep 18 16:46:46.720271 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:46:46.720248    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"volume-data-source-validator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=volume-data-source-validator pod=volume-data-source-validator-5c4db8bdf9-4bh4k_openshift-cluster-storage-operator(db1edd1b-c2ec-4a6a-b365-9305e87f8471)\"" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k" podUID="db1edd1b-c2ec-4a6a-b365-9305e87f8471"
Sep 18 16:46:47.726963 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:47.726885    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4bh4k_db1edd1b-c2ec-4a6a-b365-9305e87f8471/volume-data-source-validator/1.log"
Sep 18 16:46:55.617330 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:55.617283    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:46:59.272089 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:59.272056    2554 scope.go:117] "RemoveContainer" containerID="3d6f76e3523637046ad700de6552ccd3290cb5f10706b5fca29e8dfe746c301a"
Sep 18 16:46:59.770100 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:46:59.770041    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded" interval="6.4s"
Sep 18 16:46:59.787320 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:59.787295    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4bh4k_db1edd1b-c2ec-4a6a-b365-9305e87f8471/volume-data-source-validator/1.log"
Sep 18 16:46:59.787482 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:46:59.787413    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-4bh4k" event={"ID":"db1edd1b-c2ec-4a6a-b365-9305e87f8471","Type":"ContainerStarted","Data":"f9a4a887dfcf2b04968321da72076ad8469cafd81e59e5f52e83058e4fbd209f"}
Sep 18 16:47:03.923601 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:03.923559    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:47:05.616838 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:05.616796    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:47:07.223729 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:47:07.223689    2554 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods lws-controller-manager-6847d847f7-2hft6)" podUID="720642cf-540a-42d3-ab46-487cf5a95e72" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-2hft6"
Sep 18 16:47:11.699173 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:47:11.699085    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{kube-storage-version-migrator-operator-55889f7b47-44mkp.18d675f7e2fa990c  openshift-kube-storage-version-migrator-operator   9666 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-kube-storage-version-migrator-operator,Name:kube-storage-version-migrator-operator-55889f7b47-44mkp,UID:8a5a8e40-2a6b-4289-be46-bf1e50af868e,APIVersion:v1,ResourceVersion:7197,FieldPath:spec.containers{kube-storage-version-migrator-operator},},Reason:Started,Message:Started container kube-storage-version-migrator-operator,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:06:49 +0000 UTC,LastTimestamp:2026-09-18 16:43:13.330276335 +0000 UTC m=+2288.277775086,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:47:15.616619 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:15.616584    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:47:16.170943 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:47:16.170902    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:47:23.923325 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:23.923286    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:47:23.923734 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:23.923371    2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:47:23.923880 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:23.923859    2554 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"4dfb7acccf5d6dacd2c38fdec8ccfd80ee7a71b764360398fbb908d71e5efc8b"} pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" containerMessage="Container manager failed liveness probe, will be restarted"
Sep 18 16:47:23.923918 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:23.923899    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" containerID="cri-o://4dfb7acccf5d6dacd2c38fdec8ccfd80ee7a71b764360398fbb908d71e5efc8b" gracePeriod=10
Sep 18 16:47:23.923974 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:23.923931    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:47:24.910508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:24.910426    2554 generic.go:356] "Generic (PLEG): container finished" podID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerID="4dfb7acccf5d6dacd2c38fdec8ccfd80ee7a71b764360398fbb908d71e5efc8b" exitCode=1
Sep 18 16:47:24.910508 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:24.910452    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" event={"ID":"64d4aca4-7571-4c30-a7c8-b1747b3635b6","Type":"ContainerDied","Data":"4dfb7acccf5d6dacd2c38fdec8ccfd80ee7a71b764360398fbb908d71e5efc8b"}
Sep 18 16:47:24.910768 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:24.910539    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" event={"ID":"64d4aca4-7571-4c30-a7c8-b1747b3635b6","Type":"ContainerStarted","Data":"5336d3e8ac40ea7bf7540a6e768647ba4b5915028146395cbb63a75777e11082"}
Sep 18 16:47:24.910768 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:24.910565    2554 scope.go:117] "RemoveContainer" containerID="a287cde99b1f9edd88637239a2e71895847839397c8e38a159f3c87456408f2d"
Sep 18 16:47:24.910768 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:24.910756    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:47:25.142382 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:47:25.142340    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:47:15Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:47:15Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:47:15Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:47:15Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-130-195.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal/status?timeout=10s\": context deadline exceeded"
Sep 18 16:47:33.172369 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:47:33.172322    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:47:35.142945 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:47:35.142901    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:47:35.919434 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:35.919389    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:47:43.923395 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:43.923349    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:47:45.143353 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:47:45.143313    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:47:45.703034 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:47:45.702938    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{dns-operator-controller-manager-7b84cd7c9c-vrmv8.18d67802b4f05421  kuadrant-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kuadrant-system,Name:dns-operator-controller-manager-7b84cd7c9c-vrmv8,UID:ad3586f9-9035-40c1-bf26-143eeb8ff319,APIVersion:v1,ResourceVersion:16928,FieldPath:spec.containers{manager},},Reason:BackOff,Message:Back-off restarting failed container manager in pod dns-operator-controller-manager-7b84cd7c9c-vrmv8_kuadrant-system(ad3586f9-9035-40c1-bf26-143eeb8ff319),Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:44:14.947267617 +0000 UTC m=+2349.894766386,LastTimestamp:2026-09-18 16:44:14.947267617 +0000 UTC m=+2349.894766386,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:47:45.919112 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:45.919064    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:47:50.173439 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:47:50.173393    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:47:55.143698 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:47:55.143650    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-195.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-130-195.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:47:55.918581 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:47:55.918543    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:48:03.923000 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:48:03.922956    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:48:05.919175 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:48:05.919117    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:48:07.174584 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:48:07.174518    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-130-195.ec2.internal?timeout=10s\": context deadline exceeded" interval="7s"
Sep 18 16:48:07.226328 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:48:07.226287    2554 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57)" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:48:15.918432 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:48:15.918390    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:48:19.706125 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:48:19.705986    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57.18d67638f01928bc  opendatahub   13353 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57,UID:64d4aca4-7571-4c30-a7c8-b1747b3635b6,APIVersion:v1,ResourceVersion:13098,FieldPath:spec.containers{manager},},Reason:Pulling,Message:Pulling image \"quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2\",Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:11:28 +0000 UTC,LastTimestamp:2026-09-18 16:44:14.949562593 +0000 UTC m=+2349.897061346,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:48:23.922950 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:48:23.922911    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:48:23.923346 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:48:23.922995    2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:48:23.923489 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:48:23.923468    2554 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"5336d3e8ac40ea7bf7540a6e768647ba4b5915028146395cbb63a75777e11082"} pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" containerMessage="Container manager failed liveness probe, will be restarted"
Sep 18 16:48:23.923592 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:48:23.923509    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" containerID="cri-o://5336d3e8ac40ea7bf7540a6e768647ba4b5915028146395cbb63a75777e11082" gracePeriod=10
Sep 18 16:48:23.923683 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:48:23.923588    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Sep 18 16:48:24.043143 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:48:24.043105    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57_opendatahub(64d4aca4-7571-4c30-a7c8-b1747b3635b6)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6"
Sep 18 16:48:24.209479 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:48:24.209450    2554 generic.go:356] "Generic (PLEG): container finished" podID="64d4aca4-7571-4c30-a7c8-b1747b3635b6" containerID="5336d3e8ac40ea7bf7540a6e768647ba4b5915028146395cbb63a75777e11082" exitCode=1
Sep 18 16:48:24.209660 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:48:24.209497    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" event={"ID":"64d4aca4-7571-4c30-a7c8-b1747b3635b6","Type":"ContainerDied","Data":"5336d3e8ac40ea7bf7540a6e768647ba4b5915028146395cbb63a75777e11082"}
Sep 18 16:48:24.209660 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:48:24.209559    2554 scope.go:117] "RemoveContainer" containerID="4dfb7acccf5d6dacd2c38fdec8ccfd80ee7a71b764360398fbb908d71e5efc8b"
Sep 18 16:48:24.209917 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:48:24.209898    2554 scope.go:117] "RemoveContainer" containerID="5336d3e8ac40ea7bf7540a6e768647ba4b5915028146395cbb63a75777e11082"
Sep 18 16:48:24.210092 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:48:24.210073    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57_opendatahub(64d4aca4-7571-4c30-a7c8-b1747b3635b6)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6"
Sep 18 16:48:37.272085 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:48:37.272050    2554 scope.go:117] "RemoveContainer" containerID="5336d3e8ac40ea7bf7540a6e768647ba4b5915028146395cbb63a75777e11082"
Sep 18 16:48:37.272733 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:48:37.272234    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57_opendatahub(64d4aca4-7571-4c30-a7c8-b1747b3635b6)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6"
Sep 18 16:48:51.272295 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:48:51.272256    2554 scope.go:117] "RemoveContainer" containerID="5336d3e8ac40ea7bf7540a6e768647ba4b5915028146395cbb63a75777e11082"
Sep 18 16:48:51.272775 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:48:51.272490    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57_opendatahub(64d4aca4-7571-4c30-a7c8-b1747b3635b6)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6"
Sep 18 16:48:53.708672 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:48:53.708564    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57.18d67802cb5525a6  opendatahub    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57,UID:64d4aca4-7571-4c30-a7c8-b1747b3635b6,APIVersion:v1,ResourceVersion:13098,FieldPath:spec.containers{manager},},Reason:Pulled,Message:Successfully pulled image \"quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2\" in 373ms (373ms including waiting). Image size: 293036549 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:44:15.322973606 +0000 UTC m=+2350.270472379,LastTimestamp:2026-09-18 16:44:15.322973606 +0000 UTC m=+2350.270472379,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:49:03.727081 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:03.727052    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:49:05.271360 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:05.271327    2554 scope.go:117] "RemoveContainer" containerID="5336d3e8ac40ea7bf7540a6e768647ba4b5915028146395cbb63a75777e11082"
Sep 18 16:49:05.271833 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:49:05.271616    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57_opendatahub(64d4aca4-7571-4c30-a7c8-b1747b3635b6)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6"
Sep 18 16:49:07.230436 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:49:07.230400    2554 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods dns-operator-controller-manager-7b84cd7c9c-vrmv8)" podUID="ad3586f9-9035-40c1-bf26-143eeb8ff319" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vrmv8"
Sep 18 16:49:17.272253 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:17.272220    2554 scope.go:117] "RemoveContainer" containerID="5336d3e8ac40ea7bf7540a6e768647ba4b5915028146395cbb63a75777e11082"
Sep 18 16:49:17.272708 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:49:17.272432    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57_opendatahub(64d4aca4-7571-4c30-a7c8-b1747b3635b6)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6"
Sep 18 16:49:27.711565 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:49:27.711449    2554 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57.18d676390c24906b  opendatahub   13370 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57,UID:64d4aca4-7571-4c30-a7c8-b1747b3635b6,APIVersion:v1,ResourceVersion:13098,FieldPath:spec.containers{manager},},Reason:Created,Message:Created container: manager,Source:EventSource{Component:kubelet,Host:ip-10-0-130-195.ec2.internal,},FirstTimestamp:2026-09-18 16:11:29 +0000 UTC,LastTimestamp:2026-09-18 16:44:15.388477195 +0000 UTC m=+2350.335975972,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-195.ec2.internal,}"
Sep 18 16:49:28.692111 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:28.692079    2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:49:28.986592 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:28.986496    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6z2h8\""
Sep 18 16:49:28.999000 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:28.998977    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 18 16:49:29.014701 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.014679    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hvq9z\""
Sep 18 16:49:29.021199 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.021183    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xhcpl\""
Sep 18 16:49:29.043564 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.043541    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 16:49:29.043706 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.043687    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cduku6kld38gr\""
Sep 18 16:49:29.043750 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.043680    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 16:49:29.044348 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.044336    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 16:49:29.044542 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.044510    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6qnc4\""
Sep 18 16:49:29.049224 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.049211    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 16:49:29.089160 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.089140    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h4v9d\""
Sep 18 16:49:29.099878 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.099860    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mdh8x\""
Sep 18 16:49:29.107095 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.107074    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pvs9f\""
Sep 18 16:49:29.107146 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.107103    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-m977t\""
Sep 18 16:49:29.126286 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.126269    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 16:49:29.126711 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.126698    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 16:49:29.127023 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.127011    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 16:49:29.143677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.143659    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j9g75\""
Sep 18 16:49:29.149549 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.149516    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 16:49:29.186323 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.186303    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 18 16:49:29.186408 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.186308    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 18 16:49:29.190044 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.190030    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bgqnj\""
Sep 18 16:49:29.203299 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.203282    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 18 16:49:29.221035 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.221016    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bm6xr\""
Sep 18 16:49:29.251902 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.251854    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-69r9k\""
Sep 18 16:49:29.271627 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.271610    2554 scope.go:117] "RemoveContainer" containerID="5336d3e8ac40ea7bf7540a6e768647ba4b5915028146395cbb63a75777e11082"
Sep 18 16:49:29.271811 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:49:29.271792    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57_opendatahub(64d4aca4-7571-4c30-a7c8-b1747b3635b6)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6"
Sep 18 16:49:29.276625 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.276605    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 16:49:29.276625 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.276614    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 18 16:49:29.288646 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.288621    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 18 16:49:29.289171 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.289156    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 16:49:29.293571 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.293553    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-11725e-serving-cert\""
Sep 18 16:49:29.300014 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.299992    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 16:49:29.328338 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.328318    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 16:49:29.349593 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.349574    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kcclj\""
Sep 18 16:49:29.375301 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.375280    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 16:49:29.377473 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.377457    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 16:49:29.379895 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.379881    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 16:49:29.380842 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.380828    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mvfv2\""
Sep 18 16:49:29.390705 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.390686    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 16:49:29.406221 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.406199    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 16:49:29.415487 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.415464    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89djg\""
Sep 18 16:49:29.415679 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.415663    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kgw4g\""
Sep 18 16:49:29.421493 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.421478    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 16:49:29.422380 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.422365    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 16:49:29.461194 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.461177    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 16:49:29.466592 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.466571    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 18 16:49:29.472786 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.472770    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 18 16:49:29.473690 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.473673    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 16:49:29.484944 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.484928    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tww4c\""
Sep 18 16:49:29.487111 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.487098    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:49:29.492596 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.492582    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 16:49:29.496118 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.496104    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 16:49:29.507019 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.506974    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 16:49:29.509141 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.509127    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8cs6v\""
Sep 18 16:49:29.525226 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.525206    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dc6ds\""
Sep 18 16:49:29.536133 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.536115    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6nsd9\""
Sep 18 16:49:29.550078 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.550062    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 16:49:29.554345 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.554331    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 16:49:29.578754 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.578736    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ln5jh\""
Sep 18 16:49:29.586491 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.586474    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8gwjv\""
Sep 18 16:49:29.588158 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.588143    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n9smt\""
Sep 18 16:49:29.598668 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.598650    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 18 16:49:29.611070 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.611052    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 16:49:29.612498 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.612484    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 16:49:29.627062 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.627040    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 18 16:49:29.637492 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.637477    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 16:49:29.638880 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.638865    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 18 16:49:29.661814 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.661792    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e8mjiuub8k25v\""
Sep 18 16:49:29.662000 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.661986    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:49:29.675041 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.675024    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sr7qq\""
Sep 18 16:49:29.680905 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.680886    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c2hwb\""
Sep 18 16:49:29.681025 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.681013    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 16:49:29.685108 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.685091    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 16:49:29.694003 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.693990    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 16:49:29.698471 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.698453    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-a75117-dockercfg-lqq85\""
Sep 18 16:49:29.708128 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.708107    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vx59n\""
Sep 18 16:49:29.726576 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.726557    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 16:49:29.739493 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.739468    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 16:49:29.739616 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.739579    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rnhrt\""
Sep 18 16:49:29.741329 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.741314    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mnksp\""
Sep 18 16:49:29.750622 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.750607    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tjzzn\""
Sep 18 16:49:29.753412 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.753400    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 18 16:49:29.753801 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.753788    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-hqfkr\""
Sep 18 16:49:29.804908 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:29.804833    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mdlxq\""
Sep 18 16:49:42.272075 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:42.272041    2554 scope.go:117] "RemoveContainer" containerID="5336d3e8ac40ea7bf7540a6e768647ba4b5915028146395cbb63a75777e11082"
Sep 18 16:49:42.272614 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:49:42.272227    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57_opendatahub(64d4aca4-7571-4c30-a7c8-b1747b3635b6)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" podUID="64d4aca4-7571-4c30-a7c8-b1747b3635b6"
Sep 18 16:49:57.273806 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:57.273730    2554 scope.go:117] "RemoveContainer" containerID="5336d3e8ac40ea7bf7540a6e768647ba4b5915028146395cbb63a75777e11082"
Sep 18 16:49:57.274188 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:57.274048    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:49:58.663495 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:58.663453    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57" event={"ID":"64d4aca4-7571-4c30-a7c8-b1747b3635b6","Type":"ContainerStarted","Data":"5ad4562b34a0037e4ca9772e1c598f82eb9d1873e7269db53d41c3cb53b4f5e1"}
Sep 18 16:49:58.664010 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:49:58.663832    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:50:07.408857 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:07.408829    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-44mkp_8a5a8e40-2a6b-4289-be46-bf1e50af868e/kube-storage-version-migrator-operator/2.log"
Sep 18 16:50:07.409657 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:07.409637    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4bh4k_db1edd1b-c2ec-4a6a-b365-9305e87f8471/volume-data-source-validator/1.log"
Sep 18 16:50:07.424971 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:07.424947    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-44mkp_8a5a8e40-2a6b-4289-be46-bf1e50af868e/kube-storage-version-migrator-operator/2.log"
Sep 18 16:50:07.425677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:07.425653    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4bh4k_db1edd1b-c2ec-4a6a-b365-9305e87f8471/volume-data-source-validator/1.log"
Sep 18 16:50:09.670933 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:09.670906    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57"
Sep 18 16:50:10.628886 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.628857    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829165-6pz6k"]
Sep 18 16:50:10.629707 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629670    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60ad33d7-6cf0-4bc7-ac01-423926cf2b1b" containerName="payload-pre-processing"
Sep 18 16:50:10.629707 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629698    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="60ad33d7-6cf0-4bc7-ac01-423926cf2b1b" containerName="payload-pre-processing"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629723    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9953f9be-cbc3-44a4-a29d-4a301e468844" containerName="istio-proxy"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629732    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9953f9be-cbc3-44a4-a29d-4a301e468844" containerName="istio-proxy"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629744    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb6fd0b-b70a-4e35-804f-030e67d4bf69" containerName="istio-proxy"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629752    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb6fd0b-b70a-4e35-804f-030e67d4bf69" containerName="istio-proxy"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629762    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbebcbf4-fa27-4811-9cad-5ab137b70841" containerName="maas-api"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629769    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbebcbf4-fa27-4811-9cad-5ab137b70841" containerName="maas-api"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629781    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04710d80-2eff-4dcc-af26-51c99ef34331" containerName="payload-processing"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629789    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="04710d80-2eff-4dcc-af26-51c99ef34331" containerName="payload-processing"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629801    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58dbe8aa-7537-4391-8486-981345f7c0b5" containerName="payload-pre-processing"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629809    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="58dbe8aa-7537-4391-8486-981345f7c0b5" containerName="payload-pre-processing"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629824    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05a57950-a7f7-424c-8b42-542567916e3b" containerName="istio-proxy"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629832    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="05a57950-a7f7-424c-8b42-542567916e3b" containerName="istio-proxy"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629848    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9953f9be-cbc3-44a4-a29d-4a301e468844" containerName="istio-proxy"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629856    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9953f9be-cbc3-44a4-a29d-4a301e468844" containerName="istio-proxy"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629869    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05a57950-a7f7-424c-8b42-542567916e3b" containerName="istio-proxy"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629876    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="05a57950-a7f7-424c-8b42-542567916e3b" containerName="istio-proxy"
Sep 18 16:50:10.629881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629885    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e82c812-598f-46d3-9c31-a52fb6a37b92" containerName="payload-pre-processing"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629893    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e82c812-598f-46d3-9c31-a52fb6a37b92" containerName="payload-pre-processing"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629902    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f35b8793-64a1-4d07-801e-f9aa31da7fc2" containerName="payload-pre-processing"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629910    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f35b8793-64a1-4d07-801e-f9aa31da7fc2" containerName="payload-pre-processing"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629921    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8" containerName="main"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629928    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8" containerName="main"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629939    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56f6471d-0cdc-4796-92f9-33e223906222" containerName="main"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629947    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="56f6471d-0cdc-4796-92f9-33e223906222" containerName="main"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629962    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb" containerName="maas-api"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.629969    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb" containerName="maas-api"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.630063    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f35b8793-64a1-4d07-801e-f9aa31da7fc2" containerName="payload-pre-processing"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.630076    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbebcbf4-fa27-4811-9cad-5ab137b70841" containerName="maas-api"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.630084    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecb6fd0b-b70a-4e35-804f-030e67d4bf69" containerName="istio-proxy"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.630093    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="05a57950-a7f7-424c-8b42-542567916e3b" containerName="istio-proxy"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.630101    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="04710d80-2eff-4dcc-af26-51c99ef34331" containerName="payload-processing"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.630112    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="60ad33d7-6cf0-4bc7-ac01-423926cf2b1b" containerName="payload-pre-processing"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.630122    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9953f9be-cbc3-44a4-a29d-4a301e468844" containerName="istio-proxy"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.630130    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="58dbe8aa-7537-4391-8486-981345f7c0b5" containerName="payload-pre-processing"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.630138    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e82c812-598f-46d3-9c31-a52fb6a37b92" containerName="payload-pre-processing"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.630149    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="52502d4c-2ac8-4b53-8c2f-1c6ed2a367bb" containerName="maas-api"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.630158    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="56f6471d-0cdc-4796-92f9-33e223906222" containerName="main"
Sep 18 16:50:10.630484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.630165    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa5bd72f-b3fc-40f1-b9ce-eeaea9b802c8" containerName="main"
Sep 18 16:50:10.633386 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.633364    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829165-6pz6k"
Sep 18 16:50:10.637447 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.637425    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:50:10.643769 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.643749    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829165-6pz6k"
Sep 18 16:50:10.648008 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.647798    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829165-6pz6k"]
Sep 18 16:50:10.938622 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:10.938584    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829165-6pz6k"]
Sep 18 16:50:10.942855 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:50:10.942827    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd981b5f0_ed1f_489a_a300_2a02f8e26c48.slice/crio-8af7dc74d9f96de8d8c08baaf3d068f1d47037b2dc1709081c759af59b842d6e WatchSource:0}: Error finding container 8af7dc74d9f96de8d8c08baaf3d068f1d47037b2dc1709081c759af59b842d6e: Status 404 returned error can't find the container with id 8af7dc74d9f96de8d8c08baaf3d068f1d47037b2dc1709081c759af59b842d6e
Sep 18 16:50:11.727082 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:11.727040    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829165-6pz6k" event={"ID":"d981b5f0-ed1f-489a-a300-2a02f8e26c48","Type":"ContainerStarted","Data":"ec9db0e6775816789e05ed01f2f4f7c5704541e3b8c8c4ba776f89f5fe4e92a7"}
Sep 18 16:50:11.727269 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:11.727104    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829165-6pz6k" event={"ID":"d981b5f0-ed1f-489a-a300-2a02f8e26c48","Type":"ContainerStarted","Data":"8af7dc74d9f96de8d8c08baaf3d068f1d47037b2dc1709081c759af59b842d6e"}
Sep 18 16:50:11.752065 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:11.752022    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829165-6pz6k" podStartSLOduration=1.75200768 podStartE2EDuration="1.75200768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:50:11.751069778 +0000 UTC m=+2706.698568551" watchObservedRunningTime="2026-09-18 16:50:11.75200768 +0000 UTC m=+2706.699506824"
Sep 18 16:50:12.734163 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:12.734123    2554 generic.go:356] "Generic (PLEG): container finished" podID="d981b5f0-ed1f-489a-a300-2a02f8e26c48" containerID="ec9db0e6775816789e05ed01f2f4f7c5704541e3b8c8c4ba776f89f5fe4e92a7" exitCode=0
Sep 18 16:50:12.734589 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:12.734206    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829165-6pz6k" event={"ID":"d981b5f0-ed1f-489a-a300-2a02f8e26c48","Type":"ContainerDied","Data":"ec9db0e6775816789e05ed01f2f4f7c5704541e3b8c8c4ba776f89f5fe4e92a7"}
Sep 18 16:50:13.859998 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:13.859971    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829165-6pz6k"
Sep 18 16:50:14.748053 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:14.748022    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829165-6pz6k"
Sep 18 16:50:14.748053 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:14.748031    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829165-6pz6k" event={"ID":"d981b5f0-ed1f-489a-a300-2a02f8e26c48","Type":"ContainerDied","Data":"8af7dc74d9f96de8d8c08baaf3d068f1d47037b2dc1709081c759af59b842d6e"}
Sep 18 16:50:14.748265 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:50:14.748078    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8af7dc74d9f96de8d8c08baaf3d068f1d47037b2dc1709081c759af59b842d6e"
Sep 18 16:52:36.684435 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.684391    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-s6mm7"]
Sep 18 16:52:36.685294 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.685271    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d981b5f0-ed1f-489a-a300-2a02f8e26c48" containerName="cleanup"
Sep 18 16:52:36.685378 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.685299    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d981b5f0-ed1f-489a-a300-2a02f8e26c48" containerName="cleanup"
Sep 18 16:52:36.685378 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.685343    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb6fd0b-b70a-4e35-804f-030e67d4bf69" containerName="istio-proxy"
Sep 18 16:52:36.685378 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.685352    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb6fd0b-b70a-4e35-804f-030e67d4bf69" containerName="istio-proxy"
Sep 18 16:52:36.685567 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.685438    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9953f9be-cbc3-44a4-a29d-4a301e468844" containerName="istio-proxy"
Sep 18 16:52:36.685567 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.685457    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecb6fd0b-b70a-4e35-804f-030e67d4bf69" containerName="istio-proxy"
Sep 18 16:52:36.685567 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.685467    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="05a57950-a7f7-424c-8b42-542567916e3b" containerName="istio-proxy"
Sep 18 16:52:36.685567 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.685476    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d981b5f0-ed1f-489a-a300-2a02f8e26c48" containerName="cleanup"
Sep 18 16:52:36.688883 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.688861    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:52:36.692359 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.692339    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-sysctl-allowlist\""
Sep 18 16:52:36.787431 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.787392    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/0612fae1-f38f-4c3b-a283-654c4de55964-ready\") pod \"cni-sysctl-allowlist-ds-s6mm7\" (UID: \"0612fae1-f38f-4c3b-a283-654c4de55964\") " pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:52:36.787431 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.787432    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0612fae1-f38f-4c3b-a283-654c4de55964-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-s6mm7\" (UID: \"0612fae1-f38f-4c3b-a283-654c4de55964\") " pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:52:36.787694 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.787474    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdlpq\" (UniqueName: \"kubernetes.io/projected/0612fae1-f38f-4c3b-a283-654c4de55964-kube-api-access-sdlpq\") pod \"cni-sysctl-allowlist-ds-s6mm7\" (UID: \"0612fae1-f38f-4c3b-a283-654c4de55964\") " pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:52:36.787694 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.787557    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0612fae1-f38f-4c3b-a283-654c4de55964-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-s6mm7\" (UID: \"0612fae1-f38f-4c3b-a283-654c4de55964\") " pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:52:36.888046 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.888012    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/0612fae1-f38f-4c3b-a283-654c4de55964-ready\") pod \"cni-sysctl-allowlist-ds-s6mm7\" (UID: \"0612fae1-f38f-4c3b-a283-654c4de55964\") " pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:52:36.888046 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.888044    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0612fae1-f38f-4c3b-a283-654c4de55964-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-s6mm7\" (UID: \"0612fae1-f38f-4c3b-a283-654c4de55964\") " pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:52:36.888246 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.888075    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdlpq\" (UniqueName: \"kubernetes.io/projected/0612fae1-f38f-4c3b-a283-654c4de55964-kube-api-access-sdlpq\") pod \"cni-sysctl-allowlist-ds-s6mm7\" (UID: \"0612fae1-f38f-4c3b-a283-654c4de55964\") " pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:52:36.888246 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.888108    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0612fae1-f38f-4c3b-a283-654c4de55964-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-s6mm7\" (UID: \"0612fae1-f38f-4c3b-a283-654c4de55964\") " pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:52:36.888246 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.888159    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0612fae1-f38f-4c3b-a283-654c4de55964-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-s6mm7\" (UID: \"0612fae1-f38f-4c3b-a283-654c4de55964\") " pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:52:36.888422 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.888402    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/0612fae1-f38f-4c3b-a283-654c4de55964-ready\") pod \"cni-sysctl-allowlist-ds-s6mm7\" (UID: \"0612fae1-f38f-4c3b-a283-654c4de55964\") " pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:52:36.888624 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.888608    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0612fae1-f38f-4c3b-a283-654c4de55964-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-s6mm7\" (UID: \"0612fae1-f38f-4c3b-a283-654c4de55964\") " pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:52:36.898568 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.898546    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdlpq\" (UniqueName: \"kubernetes.io/projected/0612fae1-f38f-4c3b-a283-654c4de55964-kube-api-access-sdlpq\") pod \"cni-sysctl-allowlist-ds-s6mm7\" (UID: \"0612fae1-f38f-4c3b-a283-654c4de55964\") " pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:52:36.998736 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:36.998648    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:52:37.012081 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:52:37.012049    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0612fae1_f38f_4c3b_a283_654c4de55964.slice/crio-a1cc7fcc9c6c83e82e8cc75c5ca2390d97127186ba2500b6f5292959f0dce5eb WatchSource:0}: Error finding container a1cc7fcc9c6c83e82e8cc75c5ca2390d97127186ba2500b6f5292959f0dce5eb: Status 404 returned error can't find the container with id a1cc7fcc9c6c83e82e8cc75c5ca2390d97127186ba2500b6f5292959f0dce5eb
Sep 18 16:52:37.469166 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:37.469114    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7" event={"ID":"0612fae1-f38f-4c3b-a283-654c4de55964","Type":"ContainerStarted","Data":"24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b"}
Sep 18 16:52:37.469330 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:37.469184    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7" event={"ID":"0612fae1-f38f-4c3b-a283-654c4de55964","Type":"ContainerStarted","Data":"a1cc7fcc9c6c83e82e8cc75c5ca2390d97127186ba2500b6f5292959f0dce5eb"}
Sep 18 16:52:37.469378 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:37.469340    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:52:37.501014 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:37.500967    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7" podStartSLOduration=1.500953087 podStartE2EDuration="1.500953087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:52:37.499281525 +0000 UTC m=+2852.446780312" watchObservedRunningTime="2026-09-18 16:52:37.500953087 +0000 UTC m=+2852.448451861"
Sep 18 16:52:38.484631 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:38.484594    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:52:39.678176 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:39.678139    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-s6mm7"]
Sep 18 16:52:40.484048 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:52:40.484001    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7" podUID="0612fae1-f38f-4c3b-a283-654c4de55964" containerName="kube-multus-additional-cni-plugins" containerID="cri-o://24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b" gracePeriod=30
Sep 18 16:52:48.475380 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:52:48.475294    2554 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:52:48.476338 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:52:48.476305    2554 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:52:48.477160 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:52:48.477131    2554 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:52:48.477221 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:52:48.477174    2554 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7" podUID="0612fae1-f38f-4c3b-a283-654c4de55964" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Sep 18 16:52:58.476116 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:52:58.476071    2554 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:52:58.478670 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:52:58.477057    2554 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:52:58.478670 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:52:58.477899    2554 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:52:58.478670 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:52:58.477937    2554 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7" podUID="0612fae1-f38f-4c3b-a283-654c4de55964" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Sep 18 16:53:08.475418 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:53:08.475375    2554 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:53:08.476372 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:53:08.476347    2554 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:53:08.477267 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:53:08.477236    2554 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:53:08.477367 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:53:08.477284    2554 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7" podUID="0612fae1-f38f-4c3b-a283-654c4de55964" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Sep 18 16:53:10.566779 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.566758    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-s6mm7_0612fae1-f38f-4c3b-a283-654c4de55964/kube-multus-additional-cni-plugins/0.log"
Sep 18 16:53:10.568869 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.568853    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:53:10.631676 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.631651    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-s6mm7_0612fae1-f38f-4c3b-a283-654c4de55964/kube-multus-additional-cni-plugins/0.log"
Sep 18 16:53:10.631872 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.631698    2554 generic.go:356] "Generic (PLEG): container finished" podID="0612fae1-f38f-4c3b-a283-654c4de55964" containerID="24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b" exitCode=137
Sep 18 16:53:10.631872 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.631751    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7"
Sep 18 16:53:10.631872 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.631790    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7" event={"ID":"0612fae1-f38f-4c3b-a283-654c4de55964","Type":"ContainerDied","Data":"24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b"}
Sep 18 16:53:10.631872 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.631836    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-s6mm7" event={"ID":"0612fae1-f38f-4c3b-a283-654c4de55964","Type":"ContainerDied","Data":"a1cc7fcc9c6c83e82e8cc75c5ca2390d97127186ba2500b6f5292959f0dce5eb"}
Sep 18 16:53:10.631872 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.631858    2554 scope.go:117] "RemoveContainer" containerID="24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b"
Sep 18 16:53:10.640682 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.640664    2554 scope.go:117] "RemoveContainer" containerID="24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b"
Sep 18 16:53:10.640990 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:53:10.640972    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b\": container with ID starting with 24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b not found: ID does not exist" containerID="24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b"
Sep 18 16:53:10.641034 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.640998    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b"} err="failed to get container status \"24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b\": rpc error: code = NotFound desc = could not find container \"24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b\": container with ID starting with 24b8d4aba416670216881b4ef1e9316798bb4bef2f2f51e9a9360c17191b668b not found: ID does not exist"
Sep 18 16:53:10.694078 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.694049    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0612fae1-f38f-4c3b-a283-654c4de55964-tuning-conf-dir\") pod \"0612fae1-f38f-4c3b-a283-654c4de55964\" (UID: \"0612fae1-f38f-4c3b-a283-654c4de55964\") "
Sep 18 16:53:10.694214 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.694095    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdlpq\" (UniqueName: \"kubernetes.io/projected/0612fae1-f38f-4c3b-a283-654c4de55964-kube-api-access-sdlpq\") pod \"0612fae1-f38f-4c3b-a283-654c4de55964\" (UID: \"0612fae1-f38f-4c3b-a283-654c4de55964\") "
Sep 18 16:53:10.694214 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.694139    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/0612fae1-f38f-4c3b-a283-654c4de55964-ready\") pod \"0612fae1-f38f-4c3b-a283-654c4de55964\" (UID: \"0612fae1-f38f-4c3b-a283-654c4de55964\") "
Sep 18 16:53:10.694214 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.694146    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/0612fae1-f38f-4c3b-a283-654c4de55964-tuning-conf-dir" (OuterVolumeSpecName: "tuning-conf-dir") pod "0612fae1-f38f-4c3b-a283-654c4de55964" (UID: "0612fae1-f38f-4c3b-a283-654c4de55964"). InnerVolumeSpecName "tuning-conf-dir". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 18 16:53:10.694214 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.694169    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0612fae1-f38f-4c3b-a283-654c4de55964-cni-sysctl-allowlist\") pod \"0612fae1-f38f-4c3b-a283-654c4de55964\" (UID: \"0612fae1-f38f-4c3b-a283-654c4de55964\") "
Sep 18 16:53:10.694439 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.694392    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0612fae1-f38f-4c3b-a283-654c4de55964-ready" (OuterVolumeSpecName: "ready") pod "0612fae1-f38f-4c3b-a283-654c4de55964" (UID: "0612fae1-f38f-4c3b-a283-654c4de55964"). InnerVolumeSpecName "ready". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:53:10.694593 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.694558    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0612fae1-f38f-4c3b-a283-654c4de55964-cni-sysctl-allowlist" (OuterVolumeSpecName: "cni-sysctl-allowlist") pod "0612fae1-f38f-4c3b-a283-654c4de55964" (UID: "0612fae1-f38f-4c3b-a283-654c4de55964"). InnerVolumeSpecName "cni-sysctl-allowlist". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:53:10.694651 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.694626    2554 reconciler_common.go:299] "Volume detached for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0612fae1-f38f-4c3b-a283-654c4de55964-tuning-conf-dir\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:53:10.694651 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.694640    2554 reconciler_common.go:299] "Volume detached for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/0612fae1-f38f-4c3b-a283-654c4de55964-ready\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:53:10.696464 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.696443    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0612fae1-f38f-4c3b-a283-654c4de55964-kube-api-access-sdlpq" (OuterVolumeSpecName: "kube-api-access-sdlpq") pod "0612fae1-f38f-4c3b-a283-654c4de55964" (UID: "0612fae1-f38f-4c3b-a283-654c4de55964"). InnerVolumeSpecName "kube-api-access-sdlpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:53:10.796148 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.796064    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdlpq\" (UniqueName: \"kubernetes.io/projected/0612fae1-f38f-4c3b-a283-654c4de55964-kube-api-access-sdlpq\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:53:10.796148 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.796094    2554 reconciler_common.go:299] "Volume detached for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0612fae1-f38f-4c3b-a283-654c4de55964-cni-sysctl-allowlist\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:53:10.955801 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.955765    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-s6mm7"]
Sep 18 16:53:10.957882 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:10.957859    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-s6mm7"]
Sep 18 16:53:11.277113 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:53:11.277076    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0612fae1-f38f-4c3b-a283-654c4de55964" path="/var/lib/kubelet/pods/0612fae1-f38f-4c3b-a283-654c4de55964/volumes"
Sep 18 16:55:07.442909 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:07.442879    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-44mkp_8a5a8e40-2a6b-4289-be46-bf1e50af868e/kube-storage-version-migrator-operator/2.log"
Sep 18 16:55:07.443662 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:07.443643    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4bh4k_db1edd1b-c2ec-4a6a-b365-9305e87f8471/volume-data-source-validator/1.log"
Sep 18 16:55:07.462266 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:07.462233    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-44mkp_8a5a8e40-2a6b-4289-be46-bf1e50af868e/kube-storage-version-migrator-operator/2.log"
Sep 18 16:55:07.462979 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:07.462961    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4bh4k_db1edd1b-c2ec-4a6a-b365-9305e87f8471/volume-data-source-validator/1.log"
Sep 18 16:55:36.919108 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:36.919067    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"]
Sep 18 16:55:36.919677 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:36.919656    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0612fae1-f38f-4c3b-a283-654c4de55964" containerName="kube-multus-additional-cni-plugins"
Sep 18 16:55:36.919784 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:36.919679    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0612fae1-f38f-4c3b-a283-654c4de55964" containerName="kube-multus-additional-cni-plugins"
Sep 18 16:55:36.919843 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:36.919812    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0612fae1-f38f-4c3b-a283-654c4de55964" containerName="kube-multus-additional-cni-plugins"
Sep 18 16:55:36.923069 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:36.923046    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:36.925727 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:36.925702    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 16:55:36.935725 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:36.935701    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"]
Sep 18 16:55:36.990147 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:36.990110    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-openshift-service-ca\") pod \"maas-api-6669c6f9b7-rwtgh\" (UID: \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:36.990147 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:36.990141    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-maas-api-tls\") pod \"maas-api-6669c6f9b7-rwtgh\" (UID: \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:36.990331 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:36.990224    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlmcn\" (UniqueName: \"kubernetes.io/projected/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-kube-api-access-wlmcn\") pod \"maas-api-6669c6f9b7-rwtgh\" (UID: \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:36.990331 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:36.990260    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-metrics-certs\") pod \"maas-api-6669c6f9b7-rwtgh\" (UID: \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:37.091099 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:37.091051    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmcn\" (UniqueName: \"kubernetes.io/projected/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-kube-api-access-wlmcn\") pod \"maas-api-6669c6f9b7-rwtgh\" (UID: \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:37.091262 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:37.091113    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-metrics-certs\") pod \"maas-api-6669c6f9b7-rwtgh\" (UID: \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:37.091262 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:37.091178    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-openshift-service-ca\") pod \"maas-api-6669c6f9b7-rwtgh\" (UID: \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:37.091262 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:37.091194    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-maas-api-tls\") pod \"maas-api-6669c6f9b7-rwtgh\" (UID: \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:37.091841 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:37.091821    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-openshift-service-ca\") pod \"maas-api-6669c6f9b7-rwtgh\" (UID: \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:37.093634 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:37.093612    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-metrics-certs\") pod \"maas-api-6669c6f9b7-rwtgh\" (UID: \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:37.093730 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:37.093648    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-maas-api-tls\") pod \"maas-api-6669c6f9b7-rwtgh\" (UID: \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:37.102095 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:37.102072    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmcn\" (UniqueName: \"kubernetes.io/projected/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-kube-api-access-wlmcn\") pod \"maas-api-6669c6f9b7-rwtgh\" (UID: \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:37.233810 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:37.233784    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:37.507375 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:37.507352    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"]
Sep 18 16:55:37.509484 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:55:37.509460    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5093c7f6_8ed3_46a6_9a8a_f08658c633c2.slice/crio-520878f15dcb5766eb162bed3017e4247cb777dcb1935220099cf1703c5ea2f4 WatchSource:0}: Error finding container 520878f15dcb5766eb162bed3017e4247cb777dcb1935220099cf1703c5ea2f4: Status 404 returned error can't find the container with id 520878f15dcb5766eb162bed3017e4247cb777dcb1935220099cf1703c5ea2f4
Sep 18 16:55:37.511220 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:37.511201    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:55:38.357976 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:38.357897    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh" event={"ID":"5093c7f6-8ed3-46a6-9a8a-f08658c633c2","Type":"ContainerStarted","Data":"524211c0a1b894eac838d675f1779fb6fe16f2581cca99c66c1cb6dbe23ef759"}
Sep 18 16:55:38.357976 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:38.357951    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh" event={"ID":"5093c7f6-8ed3-46a6-9a8a-f08658c633c2","Type":"ContainerStarted","Data":"520878f15dcb5766eb162bed3017e4247cb777dcb1935220099cf1703c5ea2f4"}
Sep 18 16:55:38.358370 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:38.357986    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:38.379321 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:38.379271    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh" podStartSLOduration=1.8990267749999998 podStartE2EDuration="2.37925716s" totalImagesPullingTime="480.230385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:55:38.377896154 +0000 UTC m=+3033.325394930" watchObservedRunningTime="2026-09-18 16:55:38.37925716 +0000 UTC m=+3033.326755997"
Sep 18 16:55:44.369281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:44.369249    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:45.084194 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.084161    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"]
Sep 18 16:55:45.087848 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.087831    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"
Sep 18 16:55:45.102903 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.102870    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"]
Sep 18 16:55:45.162963 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.162928    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bda2db3-c295-4a9b-9d4f-f9f053a79a7f-openshift-service-ca\") pod \"maas-api-5b84f89b99-w4fg7\" (UID: \"2bda2db3-c295-4a9b-9d4f-f9f053a79a7f\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"
Sep 18 16:55:45.163137 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.162970    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6vnf\" (UniqueName: \"kubernetes.io/projected/2bda2db3-c295-4a9b-9d4f-f9f053a79a7f-kube-api-access-r6vnf\") pod \"maas-api-5b84f89b99-w4fg7\" (UID: \"2bda2db3-c295-4a9b-9d4f-f9f053a79a7f\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"
Sep 18 16:55:45.163137 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.163030    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bda2db3-c295-4a9b-9d4f-f9f053a79a7f-metrics-certs\") pod \"maas-api-5b84f89b99-w4fg7\" (UID: \"2bda2db3-c295-4a9b-9d4f-f9f053a79a7f\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"
Sep 18 16:55:45.163137 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.163081    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bda2db3-c295-4a9b-9d4f-f9f053a79a7f-maas-api-tls\") pod \"maas-api-5b84f89b99-w4fg7\" (UID: \"2bda2db3-c295-4a9b-9d4f-f9f053a79a7f\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"
Sep 18 16:55:45.263813 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.263771    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bda2db3-c295-4a9b-9d4f-f9f053a79a7f-maas-api-tls\") pod \"maas-api-5b84f89b99-w4fg7\" (UID: \"2bda2db3-c295-4a9b-9d4f-f9f053a79a7f\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"
Sep 18 16:55:45.263993 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.263876    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bda2db3-c295-4a9b-9d4f-f9f053a79a7f-openshift-service-ca\") pod \"maas-api-5b84f89b99-w4fg7\" (UID: \"2bda2db3-c295-4a9b-9d4f-f9f053a79a7f\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"
Sep 18 16:55:45.263993 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.263905    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6vnf\" (UniqueName: \"kubernetes.io/projected/2bda2db3-c295-4a9b-9d4f-f9f053a79a7f-kube-api-access-r6vnf\") pod \"maas-api-5b84f89b99-w4fg7\" (UID: \"2bda2db3-c295-4a9b-9d4f-f9f053a79a7f\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"
Sep 18 16:55:45.263993 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.263943    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bda2db3-c295-4a9b-9d4f-f9f053a79a7f-metrics-certs\") pod \"maas-api-5b84f89b99-w4fg7\" (UID: \"2bda2db3-c295-4a9b-9d4f-f9f053a79a7f\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"
Sep 18 16:55:45.264614 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.264589    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bda2db3-c295-4a9b-9d4f-f9f053a79a7f-openshift-service-ca\") pod \"maas-api-5b84f89b99-w4fg7\" (UID: \"2bda2db3-c295-4a9b-9d4f-f9f053a79a7f\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"
Sep 18 16:55:45.266576 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.266546    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bda2db3-c295-4a9b-9d4f-f9f053a79a7f-maas-api-tls\") pod \"maas-api-5b84f89b99-w4fg7\" (UID: \"2bda2db3-c295-4a9b-9d4f-f9f053a79a7f\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"
Sep 18 16:55:45.266671 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.266592    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bda2db3-c295-4a9b-9d4f-f9f053a79a7f-metrics-certs\") pod \"maas-api-5b84f89b99-w4fg7\" (UID: \"2bda2db3-c295-4a9b-9d4f-f9f053a79a7f\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"
Sep 18 16:55:45.273662 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.273635    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6vnf\" (UniqueName: \"kubernetes.io/projected/2bda2db3-c295-4a9b-9d4f-f9f053a79a7f-kube-api-access-r6vnf\") pod \"maas-api-5b84f89b99-w4fg7\" (UID: \"2bda2db3-c295-4a9b-9d4f-f9f053a79a7f\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"
Sep 18 16:55:45.399077 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.398990    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"
Sep 18 16:55:45.681253 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:45.681224    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"]
Sep 18 16:55:45.683629 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:55:45.683599    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bda2db3_c295_4a9b_9d4f_f9f053a79a7f.slice/crio-5e7425b299c4115b337544dcb7b2bb1ce69185a88adc6a6b08900ce52182f613 WatchSource:0}: Error finding container 5e7425b299c4115b337544dcb7b2bb1ce69185a88adc6a6b08900ce52182f613: Status 404 returned error can't find the container with id 5e7425b299c4115b337544dcb7b2bb1ce69185a88adc6a6b08900ce52182f613
Sep 18 16:55:46.400385 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:46.400288    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7" event={"ID":"2bda2db3-c295-4a9b-9d4f-f9f053a79a7f","Type":"ContainerStarted","Data":"f401233f92822115e0f8f22fb4187f8af2d5b4626727e04c3d0a745b54ac1459"}
Sep 18 16:55:46.400385 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:46.400346    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"
Sep 18 16:55:46.400385 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:46.400362    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7" event={"ID":"2bda2db3-c295-4a9b-9d4f-f9f053a79a7f","Type":"ContainerStarted","Data":"5e7425b299c4115b337544dcb7b2bb1ce69185a88adc6a6b08900ce52182f613"}
Sep 18 16:55:46.422753 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:46.422705    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7" podStartSLOduration=1.086793125 podStartE2EDuration="1.422689045s" totalImagesPullingTime="335.89592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:55:46.420571513 +0000 UTC m=+3041.368070287" watchObservedRunningTime="2026-09-18 16:55:46.422689045 +0000 UTC m=+3041.370187820"
Sep 18 16:55:52.412213 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:52.412130    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5b84f89b99-w4fg7"
Sep 18 16:55:52.484605 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:52.484571    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"]
Sep 18 16:55:52.484869 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:52.484817    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh" podUID="5093c7f6-8ed3-46a6-9a8a-f08658c633c2" containerName="maas-api" containerID="cri-o://524211c0a1b894eac838d675f1779fb6fe16f2581cca99c66c1cb6dbe23ef759" gracePeriod=30
Sep 18 16:55:52.728752 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:52.728724    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:52.832834 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:52.832796    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-metrics-certs\") pod \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\" (UID: \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\") "
Sep 18 16:55:52.833025 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:52.832905    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlmcn\" (UniqueName: \"kubernetes.io/projected/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-kube-api-access-wlmcn\") pod \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\" (UID: \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\") "
Sep 18 16:55:52.833025 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:52.832938    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-openshift-service-ca\") pod \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\" (UID: \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\") "
Sep 18 16:55:52.833025 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:52.832980    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-maas-api-tls\") pod \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\" (UID: \"5093c7f6-8ed3-46a6-9a8a-f08658c633c2\") "
Sep 18 16:55:52.833302 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:52.833278    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5093c7f6-8ed3-46a6-9a8a-f08658c633c2" (UID: "5093c7f6-8ed3-46a6-9a8a-f08658c633c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:55:52.835278 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:52.835255    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5093c7f6-8ed3-46a6-9a8a-f08658c633c2" (UID: "5093c7f6-8ed3-46a6-9a8a-f08658c633c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:55:52.835374 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:52.835284    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-kube-api-access-wlmcn" (OuterVolumeSpecName: "kube-api-access-wlmcn") pod "5093c7f6-8ed3-46a6-9a8a-f08658c633c2" (UID: "5093c7f6-8ed3-46a6-9a8a-f08658c633c2"). InnerVolumeSpecName "kube-api-access-wlmcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:55:52.835374 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:52.835278    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5093c7f6-8ed3-46a6-9a8a-f08658c633c2" (UID: "5093c7f6-8ed3-46a6-9a8a-f08658c633c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:55:52.934038 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:52.933994    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlmcn\" (UniqueName: \"kubernetes.io/projected/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-kube-api-access-wlmcn\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:55:52.934038 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:52.934028    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-openshift-service-ca\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:55:52.934038 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:52.934038    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-maas-api-tls\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:55:52.934038 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:52.934046    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5093c7f6-8ed3-46a6-9a8a-f08658c633c2-metrics-certs\") on node \"ip-10-0-130-195.ec2.internal\" DevicePath \"\""
Sep 18 16:55:53.441210 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:53.441177    2554 generic.go:356] "Generic (PLEG): container finished" podID="5093c7f6-8ed3-46a6-9a8a-f08658c633c2" containerID="524211c0a1b894eac838d675f1779fb6fe16f2581cca99c66c1cb6dbe23ef759" exitCode=0
Sep 18 16:55:53.441731 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:53.441224    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"
Sep 18 16:55:53.441731 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:53.441266    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh" event={"ID":"5093c7f6-8ed3-46a6-9a8a-f08658c633c2","Type":"ContainerDied","Data":"524211c0a1b894eac838d675f1779fb6fe16f2581cca99c66c1cb6dbe23ef759"}
Sep 18 16:55:53.441731 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:53.441331    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh" event={"ID":"5093c7f6-8ed3-46a6-9a8a-f08658c633c2","Type":"ContainerDied","Data":"520878f15dcb5766eb162bed3017e4247cb777dcb1935220099cf1703c5ea2f4"}
Sep 18 16:55:53.441731 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:53.441366    2554 scope.go:117] "RemoveContainer" containerID="524211c0a1b894eac838d675f1779fb6fe16f2581cca99c66c1cb6dbe23ef759"
Sep 18 16:55:53.450292 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:53.450274    2554 scope.go:117] "RemoveContainer" containerID="524211c0a1b894eac838d675f1779fb6fe16f2581cca99c66c1cb6dbe23ef759"
Sep 18 16:55:53.450603 ip-10-0-130-195 kubenswrapper[2554]: E0918 16:55:53.450583    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524211c0a1b894eac838d675f1779fb6fe16f2581cca99c66c1cb6dbe23ef759\": container with ID starting with 524211c0a1b894eac838d675f1779fb6fe16f2581cca99c66c1cb6dbe23ef759 not found: ID does not exist" containerID="524211c0a1b894eac838d675f1779fb6fe16f2581cca99c66c1cb6dbe23ef759"
Sep 18 16:55:53.450693 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:53.450616    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524211c0a1b894eac838d675f1779fb6fe16f2581cca99c66c1cb6dbe23ef759"} err="failed to get container status \"524211c0a1b894eac838d675f1779fb6fe16f2581cca99c66c1cb6dbe23ef759\": rpc error: code = NotFound desc = could not find container \"524211c0a1b894eac838d675f1779fb6fe16f2581cca99c66c1cb6dbe23ef759\": container with ID starting with 524211c0a1b894eac838d675f1779fb6fe16f2581cca99c66c1cb6dbe23ef759 not found: ID does not exist"
Sep 18 16:55:53.459905 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:53.459881    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"]
Sep 18 16:55:53.462396 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:53.462374    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6669c6f9b7-rwtgh"]
Sep 18 16:55:55.277596 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:55:55.277563    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5093c7f6-8ed3-46a6-9a8a-f08658c633c2" path="/var/lib/kubelet/pods/5093c7f6-8ed3-46a6-9a8a-f08658c633c2/volumes"
Sep 18 16:56:24.407371 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:24.407344    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57_64d4aca4-7571-4c30-a7c8-b1747b3635b6/copy-manifests/0.log"
Sep 18 16:56:24.425218 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:24.425166    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57_64d4aca4-7571-4c30-a7c8-b1747b3635b6/manager/5.log"
Sep 18 16:56:24.874798 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:24.874761    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5b84f89b99-w4fg7_2bda2db3-c295-4a9b-9d4f-f9f053a79a7f/maas-api/0.log"
Sep 18 16:56:25.115922 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:25.115888    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-b-w3-11725e-6db4bb86cb-qtc8s_60bb6aff-82be-45fb-abdf-d7daec90c255/maas-api/0.log"
Sep 18 16:56:25.329840 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:25.329804    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29829165-6pz6k_d981b5f0-ed1f-489a-a300-2a02f8e26c48/cleanup/0.log"
Sep 18 16:56:25.701076 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:25.701051    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829150-4rb4v_4ba15230-a6ce-4138-818f-968f07086f41/cleanup/0.log"
Sep 18 16:56:26.976798 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:26.976769    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-vrmv8_ad3586f9-9035-40c1-bf26-143eeb8ff319/manager/3.log"
Sep 18 16:56:27.091133 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:27.091108    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-mbck9_bae5c766-ff8d-4a94-a9d3-0f091dbba057/kuadrant-console-plugin/0.log"
Sep 18 16:56:28.308793 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:28.308765    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f97b6d9fd-kq8jk_d148fb58-8dd0-4d6f-b2a1-35a54628b0ad/kube-auth-proxy/0.log"
Sep 18 16:56:29.150002 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:29.149971    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r_3bc4f966-9449-43ce-85e9-f024c59d9b8b/payload-processing/0.log"
Sep 18 16:56:29.721484 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:29.721456    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-s6twd_4da75786-59ac-4f5e-843f-cb4eba985284/main/0.log"
Sep 18 16:56:29.945288 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:29.945256    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-khw86_fb38610a-7d3c-4a60-8933-0cbe967b2d36/main/0.log"
Sep 18 16:56:30.175858 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:30.175837    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7txlk_313dd42f-6969-42f5-8fdf-a1a119aa710c/main/0.log"
Sep 18 16:56:30.283373 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:30.283349    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-mwprb_f477ec87-5a46-4d37-8010-804723ca9841/main/0.log"
Sep 18 16:56:38.616419 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:38.616387    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c8d4n_d35415a6-2fc4-4544-8b27-cd22e3a87b00/global-pull-secret-syncer/0.log"
Sep 18 16:56:38.846978 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:38.846953    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vzt5f_cf58bf07-5e75-4a2c-9168-29781769dec1/konnectivity-agent/0.log"
Sep 18 16:56:38.878206 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:38.878148    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-195.ec2.internal_f8694b5494e0c8ea7c82d6e3efeab8b4/haproxy/0.log"
Sep 18 16:56:43.502055 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:43.502031    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-vrmv8_ad3586f9-9035-40c1-bf26-143eeb8ff319/manager/2.log"
Sep 18 16:56:43.502426 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:43.502155    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-vrmv8_ad3586f9-9035-40c1-bf26-143eeb8ff319/manager/3.log"
Sep 18 16:56:43.543165 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:43.543141    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-mbck9_bae5c766-ff8d-4a94-a9d3-0f091dbba057/kuadrant-console-plugin/0.log"
Sep 18 16:56:45.795334 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:45.795307    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-pqkvc_a0fa632f-e2c2-445e-a4eb-1f37cb3047b4/monitoring-plugin/0.log"
Sep 18 16:56:45.855310 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:45.855288    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8m9fc_9e9cf752-4bea-4cc1-8b09-adb51def1591/node-exporter/0.log"
Sep 18 16:56:45.895375 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:45.895356    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8m9fc_9e9cf752-4bea-4cc1-8b09-adb51def1591/kube-rbac-proxy/0.log"
Sep 18 16:56:45.933588 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:45.933569    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8m9fc_9e9cf752-4bea-4cc1-8b09-adb51def1591/init-textfile/0.log"
Sep 18 16:56:46.218691 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.218664    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-cpfp4_18e4b877-0577-49d1-aa0c-0727ecfe14f0/kube-rbac-proxy-main/0.log"
Sep 18 16:56:46.247594 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.247571    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-cpfp4_18e4b877-0577-49d1-aa0c-0727ecfe14f0/kube-rbac-proxy-self/0.log"
Sep 18 16:56:46.284143 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.284115    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-cpfp4_18e4b877-0577-49d1-aa0c-0727ecfe14f0/openshift-state-metrics/0.log"
Sep 18 16:56:46.357405 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.357372    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc31abd9-b61b-4620-a13e-1339d175dae4/prometheus/0.log"
Sep 18 16:56:46.386299 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.386238    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc31abd9-b61b-4620-a13e-1339d175dae4/config-reloader/0.log"
Sep 18 16:56:46.419234 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.419206    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc31abd9-b61b-4620-a13e-1339d175dae4/thanos-sidecar/0.log"
Sep 18 16:56:46.455273 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.455227    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc31abd9-b61b-4620-a13e-1339d175dae4/kube-rbac-proxy-web/0.log"
Sep 18 16:56:46.501543 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.501497    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc31abd9-b61b-4620-a13e-1339d175dae4/kube-rbac-proxy/0.log"
Sep 18 16:56:46.543727 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.543700    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc31abd9-b61b-4620-a13e-1339d175dae4/kube-rbac-proxy-thanos/0.log"
Sep 18 16:56:46.598462 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.598440    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc31abd9-b61b-4620-a13e-1339d175dae4/init-config-reloader/0.log"
Sep 18 16:56:46.757033 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.757009    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59478d44bc-x578z_924ab2c9-fbfc-4492-b97b-13d6a2bb1b16/telemeter-client/0.log"
Sep 18 16:56:46.789463 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.789439    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59478d44bc-x578z_924ab2c9-fbfc-4492-b97b-13d6a2bb1b16/reload/0.log"
Sep 18 16:56:46.824112 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.824091    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59478d44bc-x578z_924ab2c9-fbfc-4492-b97b-13d6a2bb1b16/kube-rbac-proxy/0.log"
Sep 18 16:56:46.859231 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.859201    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbfcdd95b-vn26r_938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9/thanos-query/0.log"
Sep 18 16:56:46.891793 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.891766    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbfcdd95b-vn26r_938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9/kube-rbac-proxy-web/0.log"
Sep 18 16:56:46.920825 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.920800    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbfcdd95b-vn26r_938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9/kube-rbac-proxy/0.log"
Sep 18 16:56:46.960873 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:46.960851    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbfcdd95b-vn26r_938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9/prom-label-proxy/0.log"
Sep 18 16:56:47.000951 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:47.000931    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbfcdd95b-vn26r_938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9/kube-rbac-proxy-rules/0.log"
Sep 18 16:56:47.047727 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:47.047668    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5dbfcdd95b-vn26r_938c4ab9-6f14-4a8e-9bc8-09c8e8c177d9/kube-rbac-proxy-metrics/0.log"
Sep 18 16:56:48.262226 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.262191    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"]
Sep 18 16:56:48.262760 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.262741    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5093c7f6-8ed3-46a6-9a8a-f08658c633c2" containerName="maas-api"
Sep 18 16:56:48.262824 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.262762    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5093c7f6-8ed3-46a6-9a8a-f08658c633c2" containerName="maas-api"
Sep 18 16:56:48.262863 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.262855    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5093c7f6-8ed3-46a6-9a8a-f08658c633c2" containerName="maas-api"
Sep 18 16:56:48.266022 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.266003    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.271110 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.271086    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7rjqm\"/\"default-dockercfg-8knlp\""
Sep 18 16:56:48.271231 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.271091    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7rjqm\"/\"kube-root-ca.crt\""
Sep 18 16:56:48.271231 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.271145    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7rjqm\"/\"openshift-service-ca.crt\""
Sep 18 16:56:48.287963 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.287934    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"]
Sep 18 16:56:48.329179 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.329152    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4630fff5-6b01-4160-8c30-33650c246ba8-lib-modules\") pod \"perf-node-gather-daemonset-th4jt\" (UID: \"4630fff5-6b01-4160-8c30-33650c246ba8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.329280 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.329182    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4630fff5-6b01-4160-8c30-33650c246ba8-proc\") pod \"perf-node-gather-daemonset-th4jt\" (UID: \"4630fff5-6b01-4160-8c30-33650c246ba8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.329280 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.329207    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4630fff5-6b01-4160-8c30-33650c246ba8-sys\") pod \"perf-node-gather-daemonset-th4jt\" (UID: \"4630fff5-6b01-4160-8c30-33650c246ba8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.329355 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.329293    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4630fff5-6b01-4160-8c30-33650c246ba8-podres\") pod \"perf-node-gather-daemonset-th4jt\" (UID: \"4630fff5-6b01-4160-8c30-33650c246ba8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.329390 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.329361    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdfpg\" (UniqueName: \"kubernetes.io/projected/4630fff5-6b01-4160-8c30-33650c246ba8-kube-api-access-mdfpg\") pod \"perf-node-gather-daemonset-th4jt\" (UID: \"4630fff5-6b01-4160-8c30-33650c246ba8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.429897 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.429867    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4630fff5-6b01-4160-8c30-33650c246ba8-lib-modules\") pod \"perf-node-gather-daemonset-th4jt\" (UID: \"4630fff5-6b01-4160-8c30-33650c246ba8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.430056 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.429906    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4630fff5-6b01-4160-8c30-33650c246ba8-proc\") pod \"perf-node-gather-daemonset-th4jt\" (UID: \"4630fff5-6b01-4160-8c30-33650c246ba8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.430056 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.429987    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4630fff5-6b01-4160-8c30-33650c246ba8-proc\") pod \"perf-node-gather-daemonset-th4jt\" (UID: \"4630fff5-6b01-4160-8c30-33650c246ba8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.430056 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.430027    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4630fff5-6b01-4160-8c30-33650c246ba8-sys\") pod \"perf-node-gather-daemonset-th4jt\" (UID: \"4630fff5-6b01-4160-8c30-33650c246ba8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.430177 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.430053    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4630fff5-6b01-4160-8c30-33650c246ba8-lib-modules\") pod \"perf-node-gather-daemonset-th4jt\" (UID: \"4630fff5-6b01-4160-8c30-33650c246ba8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.430177 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.430109    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4630fff5-6b01-4160-8c30-33650c246ba8-podres\") pod \"perf-node-gather-daemonset-th4jt\" (UID: \"4630fff5-6b01-4160-8c30-33650c246ba8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.430177 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.430152    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdfpg\" (UniqueName: \"kubernetes.io/projected/4630fff5-6b01-4160-8c30-33650c246ba8-kube-api-access-mdfpg\") pod \"perf-node-gather-daemonset-th4jt\" (UID: \"4630fff5-6b01-4160-8c30-33650c246ba8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.430276 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.430185    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4630fff5-6b01-4160-8c30-33650c246ba8-podres\") pod \"perf-node-gather-daemonset-th4jt\" (UID: \"4630fff5-6b01-4160-8c30-33650c246ba8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.430276 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.430111    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4630fff5-6b01-4160-8c30-33650c246ba8-sys\") pod \"perf-node-gather-daemonset-th4jt\" (UID: \"4630fff5-6b01-4160-8c30-33650c246ba8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.441265 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.441246    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdfpg\" (UniqueName: \"kubernetes.io/projected/4630fff5-6b01-4160-8c30-33650c246ba8-kube-api-access-mdfpg\") pod \"perf-node-gather-daemonset-th4jt\" (UID: \"4630fff5-6b01-4160-8c30-33650c246ba8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.575776 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.575718    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:48.853418 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:48.853389    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"]
Sep 18 16:56:48.855224 ip-10-0-130-195 kubenswrapper[2554]: W0918 16:56:48.855192    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4630fff5_6b01_4160_8c30_33650c246ba8.slice/crio-29daa273e69c15a03a2ce8b370a52fd862b3460c4e9605c86eceb4d9440bf2c0 WatchSource:0}: Error finding container 29daa273e69c15a03a2ce8b370a52fd862b3460c4e9605c86eceb4d9440bf2c0: Status 404 returned error can't find the container with id 29daa273e69c15a03a2ce8b370a52fd862b3460c4e9605c86eceb4d9440bf2c0
Sep 18 16:56:49.721410 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:49.721372    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt" event={"ID":"4630fff5-6b01-4160-8c30-33650c246ba8","Type":"ContainerStarted","Data":"246e4b35a86cf72c32a7ff257afabd2b76746f24ed79d6239f04ffa07c08388a"}
Sep 18 16:56:49.721818 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:49.721426    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt" event={"ID":"4630fff5-6b01-4160-8c30-33650c246ba8","Type":"ContainerStarted","Data":"29daa273e69c15a03a2ce8b370a52fd862b3460c4e9605c86eceb4d9440bf2c0"}
Sep 18 16:56:49.721818 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:49.721506    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:49.746937 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:49.746893    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt" podStartSLOduration=1.746879262 podStartE2EDuration="1.746879262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:56:49.745764786 +0000 UTC m=+3104.693263560" watchObservedRunningTime="2026-09-18 16:56:49.746879262 +0000 UTC m=+3104.694378037"
Sep 18 16:56:50.338377 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:50.338348    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4bh4k_db1edd1b-c2ec-4a6a-b365-9305e87f8471/volume-data-source-validator/2.log"
Sep 18 16:56:50.338574 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:50.338514    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-4bh4k_db1edd1b-c2ec-4a6a-b365-9305e87f8471/volume-data-source-validator/1.log"
Sep 18 16:56:51.554024 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:51.554000    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xxw4k_cca8c0d4-a36f-4622-8f97-e18b71343f72/dns/0.log"
Sep 18 16:56:51.601881 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:51.601858    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xxw4k_cca8c0d4-a36f-4622-8f97-e18b71343f72/kube-rbac-proxy/0.log"
Sep 18 16:56:51.719622 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:51.719593    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9286k_8906b592-b7c8-46a7-ab14-c89b0803fd8d/dns-node-resolver/0.log"
Sep 18 16:56:52.507223 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:52.507184    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-26zfw_4d012abb-e750-4643-b3fe-8cffd9723e76/node-ca/0.log"
Sep 18 16:56:54.040956 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:54.040928    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f97b6d9fd-kq8jk_d148fb58-8dd0-4d6f-b2a1-35a54628b0ad/kube-auth-proxy/0.log"
Sep 18 16:56:54.288281 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:54.288252    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r_3bc4f966-9449-43ce-85e9-f024c59d9b8b/payload-processing/0.log"
Sep 18 16:56:55.322188 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:55.322153    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gsxcc_09f70d2c-590d-4474-ac55-e55056a054cd/serve-healthcheck-canary/0.log"
Sep 18 16:56:55.736923 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:55.736896    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-th4jt"
Sep 18 16:56:56.122841 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:56.122773    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-svv8w_b20bad3b-c233-42b2-8860-2a53b864ade8/kube-rbac-proxy/0.log"
Sep 18 16:56:56.153112 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:56.153084    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-svv8w_b20bad3b-c233-42b2-8860-2a53b864ade8/exporter/0.log"
Sep 18 16:56:56.193019 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:56.192966    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-svv8w_b20bad3b-c233-42b2-8860-2a53b864ade8/extractor/0.log"
Sep 18 16:56:58.791795 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:58.791749    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f97b6d9fd-kq8jk_d148fb58-8dd0-4d6f-b2a1-35a54628b0ad/kube-auth-proxy/0.log"
Sep 18 16:56:58.971769 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:58.971727    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r_3bc4f966-9449-43ce-85e9-f024c59d9b8b/payload-processing/0.log"
Sep 18 16:56:59.192448 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:59.192421    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57_64d4aca4-7571-4c30-a7c8-b1747b3635b6/manager/4.log"
Sep 18 16:56:59.207581 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:59.207558    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57_64d4aca4-7571-4c30-a7c8-b1747b3635b6/manager/5.log"
Sep 18 16:56:59.230815 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:56:59.230796    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cdc5fd7cb-ljb57_64d4aca4-7571-4c30-a7c8-b1747b3635b6/copy-manifests/0.log"
Sep 18 16:57:00.656218 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:57:00.656182    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6847d847f7-2hft6_720642cf-540a-42d3-ab46-487cf5a95e72/manager/1.log"
Sep 18 16:57:00.657937 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:57:00.657918    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6847d847f7-2hft6_720642cf-540a-42d3-ab46-487cf5a95e72/manager/2.log"
Sep 18 16:57:01.067506 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:57:01.067480    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f97b6d9fd-kq8jk_d148fb58-8dd0-4d6f-b2a1-35a54628b0ad/kube-auth-proxy/0.log"
Sep 18 16:57:01.228382 ip-10-0-130-195 kubenswrapper[2554]: I0918 16:57:01.228351    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-shared-a-w3-a75117-57fdf4779f-tn62r_3bc4f966-9449-43ce-85e9-f024c59d9b8b/payload-processing/0.log"