Sep 16 18:14:16.990738 ip-10-0-136-38 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:14:16.990750 ip-10-0-136-38 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:14:16.990757 ip-10-0-136-38 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:14:16.990906 ip-10-0-136-38 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:14:27.076738 ip-10-0-136-38 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:14:27.076761 ip-10-0-136-38 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 45a399723b4842718b35a1b9c8c245d9 --
Sep 16 18:16:16.341515 ip-10-0-136-38 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:16:16.668151 ip-10-0-136-38 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:16:16.668151 ip-10-0-136-38 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:16:16.668151 ip-10-0-136-38 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:16:16.668151 ip-10-0-136-38 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:16:16.668151 ip-10-0-136-38 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:16:16.670271 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.670171    2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:16:16.673719 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673700    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:16:16.673719 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673718    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673726    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673733    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673738    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673743    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673747    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673751    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673755    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673759    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673763    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673767    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673771    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673774    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673778    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673782    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673786    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673790    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673794    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673798    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673803    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:16:16.673859 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673808    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673812    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673816    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673820    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673824    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673828    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673833    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673837    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673841    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673845    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673849    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673853    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673857    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673862    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673866    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673870    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673874    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673878    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673882    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673887    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:16:16.674708 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673891    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673895    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673899    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673903    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673908    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673912    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673916    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673920    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673924    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673928    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673932    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673938    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673942    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673946    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673951    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673955    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673960    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673965    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673970    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673974    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:16:16.675551 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673979    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673983    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673989    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.673995    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674000    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674005    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674009    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674014    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674019    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674024    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674038    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674043    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674047    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674052    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674056    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674060    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674064    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674068    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674072    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:16:16.676413 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674076    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674081    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674086    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674090    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674097    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674103    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674107    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674112    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674116    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674121    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674125    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674129    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674133    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674138    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674142    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674146    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674151    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674155    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.674159    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:16:16.677251 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675320    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675330    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675334    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675338    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675342    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675347    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675352    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675356    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675361    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675365    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675369    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675374    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675378    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675382    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675388    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675393    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675398    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675402    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675406    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675410    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:16:16.677909 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675417    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675421    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675425    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675430    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675433    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675437    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675442    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675446    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675450    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675454    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675458    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675462    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675466    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675471    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675477    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675483    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675488    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675493    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675498    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675503    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:16:16.678436 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675507    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675511    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675516    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675520    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675524    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675528    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675534    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675538    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675542    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675546    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675550    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675554    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675558    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675563    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675567    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675570    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675575    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675579    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675583    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:16:16.678929 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675588    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675592    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675596    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675600    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675604    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675608    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675612    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675616    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675620    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675624    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675629    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675633    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675638    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675642    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675646    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675650    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675654    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675659    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675663    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675668    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:16:16.679429 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675674    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675678    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675682    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675686    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675690    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675694    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675698    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675702    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675707    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675711    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675718    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675723    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675729    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675734    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675738    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675742    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675747    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675750    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675754    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.675759    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:16:16.679928 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676374    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676382    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676387    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676391    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676396    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676401    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676405    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676409    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676413    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676416    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676421    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676425    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676429    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676432    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676436    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676440    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676445    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676449    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676453    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676457    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:16:16.680435 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676462    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676466    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676470    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676474    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676479    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676483    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676487    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676492    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676496    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676500    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676504    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676508    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676512    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676516    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676520    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676524    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676528    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676537    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676542    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676546    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:16:16.680934 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676550    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676554    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676558    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676562    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676566    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676570    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676574    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676578    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676582    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676587    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676591    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676597    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676603    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676608    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676612    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676616    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676621    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676625    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676629    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:16:16.681440 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676633    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676637    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676641    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676647    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676651    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676656    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676660    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676664    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676668    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676672    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676676    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676686    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676691    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676695    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676699    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676704    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676708    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676712    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676716    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676720    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:16:16.681902 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676724    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676728    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676738    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676744    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676749    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676754    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676758    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676764    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676770    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676775    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676780    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676785    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676790    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676794    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676798    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676803    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676807    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676811    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676815    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:16:16.682416 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.676819    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.676933    2554 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.676991    2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677001    2554 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677008    2554 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677015    2554 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677020    2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677027    2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677034    2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677039    2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677044    2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677050    2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677055    2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677060    2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677065    2554 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677072    2554 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677077    2554 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677081    2554 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677086    2554 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677092    2554 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677097    2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677102    2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677108    2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677113    2554 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:16:16.682911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677119    2554 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677124    2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677129    2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677134    2554 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677139    2554 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677143    2554 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677148    2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677153    2554 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677158    2554 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677164    2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677169    2554 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677174    2554 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677178    2554 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677183    2554 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677188    2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677195    2554 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677200    2554 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677205    2554 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677210    2554 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677214    2554 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677220    2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677225    2554 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677244    2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677251    2554 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677256    2554 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:16:16.683515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677261    2554 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677265    2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677269    2554 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677274    2554 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677278    2554 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677284    2554 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677293    2554 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677298    2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677303    2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677309    2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677314    2554 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677319    2554 flags.go:64] FLAG: --help="false"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677324    2554 flags.go:64] FLAG: --hostname-override="ip-10-0-136-38.ec2.internal"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677329    2554 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677334    2554 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677339    2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677345    2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677350    2554 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677355    2554 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677360    2554 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677364    2554 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677369    2554 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677375    2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677381    2554 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:16:16.684125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677385    2554 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677390    2554 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677395    2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677400    2554 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677404    2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677409    2554 flags.go:64] FLAG: --lock-file=""
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677415    2554 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677420    2554 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677425    2554 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677433    2554 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677438    2554 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677442    2554 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677447    2554 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677454    2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677460    2554 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677465    2554 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677469    2554 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677476    2554 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677481    2554 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677486    2554 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677491    2554 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677496    2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677501    2554 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677505    2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677510    2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:16:16.684776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677515    2554 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677520    2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677531    2554 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677536    2554 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677541    2554 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677545    2554 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677558    2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677567    2554 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677571    2554 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677577    2554 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677581    2554 flags.go:64] FLAG: --port="10250"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677586    2554 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677590    2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a7106772cd2f5ea4"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677614    2554 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677637    2554 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677643    2554 flags.go:64] FLAG: --register-node="true"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677647    2554 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677652    2554 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677655    2554 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677659    2554 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677663    2554 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677666    2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677670    2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677673    2554 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677676    2554 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:16:16.685418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677679    2554 flags.go:64] FLAG: --runonce="false"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677682    2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677686    2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677689    2554 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677692    2554 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677695    2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677699    2554 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677702    2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677705    2554 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677708    2554 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677711    2554 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677714    2554 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677716    2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677720    2554 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677723    2554 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677726    2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677734    2554 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677737    2554 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677740    2554 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677746    2554 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677749    2554 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677753    2554 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677757    2554 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677760    2554 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677763    2554 flags.go:64] FLAG: --v="2"
Sep 16 18:16:16.686026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677767    2554 flags.go:64] FLAG: --version="false"
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677771    2554 flags.go:64] FLAG: --vmodule=""
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677775    2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.677778    2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677897    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677902    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677906    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677910    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677912    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677916    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677919    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677921    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677924    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677927    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677930    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677933    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677935    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677938    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677941    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677943    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:16:16.686833 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677946    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677948    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677951    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677953    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677956    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677958    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677963    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677966    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677970    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677974    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677977    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677980    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677983    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677986    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677989    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677991    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677994    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677997    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.677999    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:16:16.687652 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678002    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678004    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678007    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678010    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678012    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678015    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678017    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678020    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678022    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678025    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678027    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678030    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678032    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678035    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678038    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678040    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678043    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678048    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678052    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678057    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:16:16.688130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678060    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678063    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678066    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678069    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678072    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678075    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678077    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678080    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678083    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678086    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678089    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678091    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678094    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678096    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678099    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678102    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678105    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678108    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678111    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678113    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:16:16.688639 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678116    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678118    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678121    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678123    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678126    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678129    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678131    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678134    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678137    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678141    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678144    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678147    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678149    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678152    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678156    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678159    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678161    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678164    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678167    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678170    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:16:16.689130 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678172    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:16:16.689735 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678175    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:16:16.689735 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678178    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:16:16.689735 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.678180    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:16:16.689735 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.678734    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:16:16.689735 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.685635    2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:16:16.689735 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.685650    2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:16:16.689735 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.685669    2554 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:16:16.689735 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.685674    2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:16:16.689735 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685745    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:16:16.689735 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685750    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:16:16.689735 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685754    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:16:16.689735 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685757    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:16:16.689735 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685760    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:16:16.689735 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685763    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:16:16.689735 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685775    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685778    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685781    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685783    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685786    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685789    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685791    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685794    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685797    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685800    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685802    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685805    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685808    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685810    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685813    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685816    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685819    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685822    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685825    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685839    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:16:16.690138 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685842    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685845    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685848    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685851    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685855    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685857    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685860    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685863    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685865    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685868    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685871    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685873    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685876    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685879    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685883    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685887    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685894    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685897    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685900    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:16:16.690682 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685903    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685905    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685908    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685911    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685913    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685916    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685919    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685921    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685926    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685930    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685933    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685936    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685939    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685942    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685945    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685948    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685951    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685954    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685956    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:16:16.691210 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685960    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685962    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685966    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685969    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685972    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685975    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685978    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685981    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685983    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685986    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685989    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685991    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685994    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685996    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.685999    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686001    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686004    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686006    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686009    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:16:16.691761 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686011    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:16:16.692221 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686014    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:16:16.692221 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686016    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:16:16.692221 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686019    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:16:16.692221 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686022    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:16:16.692221 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686025    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:16:16.692221 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686027    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:16:16.692221 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686030    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:16:16.692221 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686033    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:16:16.692221 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686037    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:16:16.692221 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686040    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:16:16.692221 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686043    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:16:16.692221 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686046    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:16:16.692221 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686049    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:16:16.692221 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686052    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:16:16.692221 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686054    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:16:16.692604 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.686059    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:16:16.692604 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686165    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:16:16.692604 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686170    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:16:16.692604 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686173    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:16:16.692604 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686176    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:16:16.692604 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686179    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:16:16.692604 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686182    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:16:16.692604 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686185    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:16:16.692604 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686188    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:16:16.692604 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686191    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:16:16.692604 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686194    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:16:16.692604 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686197    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:16:16.692604 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686200    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:16:16.692604 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686203    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:16:16.692604 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686205    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686208    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686212    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686216    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686219    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686222    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686224    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686243    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686248    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686250    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686253    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686256    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686259    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686262    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686265    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686267    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686270    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686272    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686275    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686278    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:16:16.693010 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686280    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686283    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686285    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686288    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686290    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686293    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686295    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686298    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686300    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686303    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686307    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686310    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686313    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686316    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686318    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686321    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686323    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686326    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686329    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:16:16.693511 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686331    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686335    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686338    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686340    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686343    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686346    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686348    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686351    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686353    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686356    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686359    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686361    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686364    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686366    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686369    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686371    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686374    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686376    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686379    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:16:16.694000 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686382    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686386    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686389    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686392    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686395    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686398    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686400    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686403    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686405    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686408    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686410    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686413    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686415    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686418    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686420    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686424    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686426    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686429    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686431    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686434    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:16:16.694474 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686436    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:16:16.694962 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686439    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:16:16.694962 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686441    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:16:16.694962 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686444    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:16:16.694962 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686446    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:16:16.694962 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686449    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:16:16.694962 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686451    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:16:16.694962 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:16.686454    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:16:16.694962 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.686459    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:16:16.694962 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.687194    2554 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:16:16.694962 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.689495    2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:16:16.694962 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.690328    2554 server.go:1019] "Starting client certificate rotation"
Sep 16 18:16:16.694962 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.690420    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:16:16.694962 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.690457    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:16:16.695309 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.692638    2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:16:16.695309 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.693716    2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:16:16.703429 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.703412    2554 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:16:16.707518 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.707501    2554 log.go:25] "Validated CRI v1 image API"
Sep 16 18:16:16.708268 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.708252    2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:16:16.711193 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.711172    2554 fs.go:135] Filesystem UUIDs: map[70e6b8ef-14f2-4441-a7d7-e47bfacc2560:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9b2ed71d-bfa7-460b-934b-9a560931af35:/dev/nvme0n1p4]
Sep 16 18:16:16.711278 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.711192    2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:16:16.715241 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.715121    2554 manager.go:216] Machine: {Timestamp:2026-09-16 18:16:16.714670322 +0000 UTC m=+0.273838148 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097289 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2641717b4268f147f48bc74c7d12f9 SystemUUID:ec264171-7b42-68f1-47f4-8bc74c7d12f9 BootID:45a39972-3b48-4271-8b35-a1b9c8c245d9 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:a5:40:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:a5:40:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:f0:ea:49:09:42 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:16:16.715241 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.715220    2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:16:16.715376 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.715301    2554 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:16:16.717921 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.717895    2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:16:16.718084 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.717922    2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-38.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:16:16.718344 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.718329    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:16:16.718748 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.718735    2554 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:16:16.718803 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.718755    2554 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:16:16.718803 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.718766    2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:16:16.719452 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.719440    2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:16:16.720477 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.720462    2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:16:16.723886 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.723872    2554 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:16:16.724015 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.724004    2554 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:16:16.724129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.724041    2554 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:16:16.724129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.724055    2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:16:16.724129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.724073    2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:16:16.724129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.724089    2554 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:16:16.724129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.724100    2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:16:16.724477 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.724463    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:16:16.724539 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.724487    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:16:16.728050 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.728029    2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:16:16.730548 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.730532    2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:16:16.730635 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.730570    2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:16:16.732801 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.732787    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:16:16.732867 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.732809    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:16:16.732867 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.732818    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:16:16.732867 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.732826    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:16:16.732867 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.732838    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:16:16.732867 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.732847    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:16:16.732867 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.732868    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:16:16.733082 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.732877    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:16:16.733082 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.732895    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:16:16.733082 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.732905    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:16:16.733082 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.732917    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:16:16.733082 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.732926    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:16:16.733587 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.733577    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:16:16.733638 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.733590    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:16:16.736030 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.736006    2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-38.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:16:16.736686 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:16.736660    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:16:16.736761 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:16.736660    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:16:16.737119 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.737104    2554 server.go:1268] "Started kubelet"
Sep 16 18:16:16.737187 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.737167    2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:16:16.737297 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.737256    2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:16:16.737395 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.737312    2554 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:16:16.738009 ip-10-0-136-38 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:16:16.738627 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.738608    2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:16:16.740404 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.740343    2554 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:16:16.741529 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:16.741501    2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:16:16.743688 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:16.743021    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-38.ec2.internal.18d5dfdf36a44ebc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-38.ec2.internal,UID:ip-10-0-136-38.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-38.ec2.internal,},FirstTimestamp:2026-09-16 18:16:16.737078972 +0000 UTC m=+0.296246799,LastTimestamp:2026-09-16 18:16:16.737078972 +0000 UTC m=+0.296246799,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-38.ec2.internal,}"
Sep 16 18:16:16.744494 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.744473    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:16:16.744850 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.744835    2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:16:16.744934 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:16.744924    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Sep 16 18:16:16.745353 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.745337    2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:16:16.745353 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.745338    2554 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:16:16.745479 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.745360    2554 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:16:16.745479 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.745448    2554 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:16:16.745479 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.745457    2554 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:16:16.745618 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.745600    2554 factory.go:55] Registering systemd factory
Sep 16 18:16:16.745618 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.745617    2554 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:16:16.745824 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.745812    2554 factory.go:153] Registering CRI-O factory
Sep 16 18:16:16.745824 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.745825    2554 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:16:16.745891 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.745858    2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:16:16.745891 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.745870    2554 factory.go:103] Registering Raw factory
Sep 16 18:16:16.745891 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.745880    2554 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:16:16.746143 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.746134    2554 manager.go:349] Starting recovery of all containers
Sep 16 18:16:16.752936 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:16.752908    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:16:16.754751 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:16.754722    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:16:16.757552 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.757530    2554 manager.go:354] Recovery completed
Sep 16 18:16:16.762611 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.762498    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:16:16.764321 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.764299    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:16:16.764390 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.764334    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:16:16.764390 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.764347    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:16:16.764838 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.764823    2554 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:16:16.764838 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.764832    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:16:16.764960 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.764835    2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:16:16.764960 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.764865    2554 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:16:16.765568 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.765552    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:16:16.765627 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.765579    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:16:16.765627 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.765590    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:16:16.765948 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.765931    2554 policy_none.go:49] "None policy: Start"
Sep 16 18:16:16.766011 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.765950    2554 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:16:16.766011 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.765960    2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:16:16.766327 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:16.766255    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-38.ec2.internal.18d5dfdf3843f995  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-38.ec2.internal,UID:ip-10-0-136-38.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-38.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-38.ec2.internal,},FirstTimestamp:2026-09-16 18:16:16.764320149 +0000 UTC m=+0.323487974,LastTimestamp:2026-09-16 18:16:16.764320149 +0000 UTC m=+0.323487974,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-38.ec2.internal,}"
Sep 16 18:16:16.766852 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.766841    2554 policy_none.go:47] "Start"
Sep 16 18:16:16.770682 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.770581    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cq74m"
Sep 16 18:16:16.776620 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:16.776541    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-38.ec2.internal.18d5dfdf38444bd6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-38.ec2.internal,UID:ip-10-0-136-38.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-38.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-38.ec2.internal,},FirstTimestamp:2026-09-16 18:16:16.764341206 +0000 UTC m=+0.323509032,LastTimestamp:2026-09-16 18:16:16.764341206 +0000 UTC m=+0.323509032,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-38.ec2.internal,}"
Sep 16 18:16:16.779359 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.779341    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cq74m"
Sep 16 18:16:16.790940 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.790926    2554 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:16:16.791026 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:16.790972    2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:16:16.791026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.790985    2554 server.go:87] "Starting device plugin registration server"
Sep 16 18:16:16.791296 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.791281    2554 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:16:16.791381 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.791296    2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:16:16.791483 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.791466    2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:16:16.791556 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.791539    2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:16:16.791556 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.791556    2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:16:16.791945 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:16.791927    2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:16:16.792004 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:16.791981    2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-38.ec2.internal\" not found"
Sep 16 18:16:16.875850 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.875821    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:16:16.877001 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.876975    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:16:16.877090 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.877029    2554 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:16:16.877090 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.877057    2554 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:16:16.877174 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:16.877148    2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:16:16.879457 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.879436    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:16:16.892328 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.892314    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:16:16.893195 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.893178    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:16:16.893294 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.893210    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:16:16.893294 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.893227    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:16:16.893294 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.893267    2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-38.ec2.internal"
Sep 16 18:16:16.902017 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.902000    2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-38.ec2.internal"
Sep 16 18:16:16.902078 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:16.902025    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-38.ec2.internal\": node \"ip-10-0-136-38.ec2.internal\" not found"
Sep 16 18:16:16.947848 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:16.947790    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Sep 16 18:16:16.978034 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.978013    2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"]
Sep 16 18:16:16.978122 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.978084    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:16:16.979643 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.979629    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:16:16.979708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.979656    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:16:16.979708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.979669    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:16:16.981240 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.981216    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:16:16.981363 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.981349    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Sep 16 18:16:16.981441 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.981376    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:16:16.981975 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.981960    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:16:16.982054 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.981977    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:16:16.982054 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.981988    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:16:16.982054 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.981996    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:16:16.982054 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.982002    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:16:16.982250 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.982005    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:16:16.983944 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.983929    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Sep 16 18:16:16.984020 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.983957    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:16:16.988516 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.988503    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:16:16.988601 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.988525    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:16:16.988601 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:16.988538    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:16:17.003744 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:17.003727    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-38.ec2.internal\" not found" node="ip-10-0-136-38.ec2.internal"
Sep 16 18:16:17.009560 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:17.009542    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-38.ec2.internal\" not found" node="ip-10-0-136-38.ec2.internal"
Sep 16 18:16:17.046512 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.046485    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e066247b444270d650924fbf370e196-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"5e066247b444270d650924fbf370e196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Sep 16 18:16:17.046580 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.046517    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f74f6b140c4aa2324e29421d89be3fe-config\") pod \"kube-apiserver-proxy-ip-10-0-136-38.ec2.internal\" (UID: \"0f74f6b140c4aa2324e29421d89be3fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Sep 16 18:16:17.046580 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.046538    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e066247b444270d650924fbf370e196-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"5e066247b444270d650924fbf370e196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Sep 16 18:16:17.048596 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:17.048584    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Sep 16 18:16:17.146887 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.146864    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e066247b444270d650924fbf370e196-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"5e066247b444270d650924fbf370e196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Sep 16 18:16:17.146887 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.146888    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e066247b444270d650924fbf370e196-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"5e066247b444270d650924fbf370e196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Sep 16 18:16:17.147019 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.146904    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f74f6b140c4aa2324e29421d89be3fe-config\") pod \"kube-apiserver-proxy-ip-10-0-136-38.ec2.internal\" (UID: \"0f74f6b140c4aa2324e29421d89be3fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Sep 16 18:16:17.147019 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.146931    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f74f6b140c4aa2324e29421d89be3fe-config\") pod \"kube-apiserver-proxy-ip-10-0-136-38.ec2.internal\" (UID: \"0f74f6b140c4aa2324e29421d89be3fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Sep 16 18:16:17.147019 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.146963    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e066247b444270d650924fbf370e196-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"5e066247b444270d650924fbf370e196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Sep 16 18:16:17.147019 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.147014    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e066247b444270d650924fbf370e196-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"5e066247b444270d650924fbf370e196\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Sep 16 18:16:17.148970 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:17.148954    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Sep 16 18:16:17.249755 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:17.249695    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Sep 16 18:16:17.306878 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.306844    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Sep 16 18:16:17.313083 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.313066    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Sep 16 18:16:17.350063 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:17.350043    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Sep 16 18:16:17.450563 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:17.450547    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Sep 16 18:16:17.550782 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:17.550725    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Sep 16 18:16:17.650974 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:17.650945    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Sep 16 18:16:17.690139 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.690111    2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:16:17.690780 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.690264    2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:16:17.751114 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:17.751063    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Sep 16 18:16:17.783446 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.783406    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:11:17 +0000 UTC" deadline="2028-05-12 13:17:31.31080888 +0000 UTC"
Sep 16 18:16:17.783666 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.783452    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14491h1m13.527372602s"
Sep 16 18:16:17.851553 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:17.851487    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Sep 16 18:16:17.902983 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.902963    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:16:17.903968 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.903947    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:16:17.904043 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.903991    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:16:17.904043 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.904009    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:16:17.924172 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:17.924136    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f74f6b140c4aa2324e29421d89be3fe.slice/crio-cc8df6e03a4917c31a1a3ffa6298fdd2ff6c0c7d10790fa065ebbacd1f5c8050 WatchSource:0}: Error finding container cc8df6e03a4917c31a1a3ffa6298fdd2ff6c0c7d10790fa065ebbacd1f5c8050: Status 404 returned error can't find the container with id cc8df6e03a4917c31a1a3ffa6298fdd2ff6c0c7d10790fa065ebbacd1f5c8050
Sep 16 18:16:17.925353 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:17.925331    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e066247b444270d650924fbf370e196.slice/crio-ed60c272a2009f4e206c8d57ec454bdd3da26561b4a6d02e7456a044210b1e6a WatchSource:0}: Error finding container ed60c272a2009f4e206c8d57ec454bdd3da26561b4a6d02e7456a044210b1e6a: Status 404 returned error can't find the container with id ed60c272a2009f4e206c8d57ec454bdd3da26561b4a6d02e7456a044210b1e6a
Sep 16 18:16:17.930061 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:17.930044    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:16:17.952445 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:17.952421    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Sep 16 18:16:18.036126 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.036103    2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:16:18.045105 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.045083    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Sep 16 18:16:18.060423 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.060405    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:16:18.062152 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.062139    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Sep 16 18:16:18.069620 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.069606    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:16:18.135414 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.135363    2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:16:18.281752 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.281722    2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:16:18.721829 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.721646    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:16:18.724633 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.724602    2554 apiserver.go:52] "Watching apiserver"
Sep 16 18:16:18.730176 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.730149    2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:16:18.730692 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.730654    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-7w74g","kube-system/konnectivity-agent-jtdvj","kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz","openshift-dns/node-resolver-dqj4w","openshift-multus/multus-additional-cni-plugins-v4q6m","openshift-multus/multus-blplt","openshift-network-operator/iptables-alerter-cs2z7","openshift-cluster-node-tuning-operator/tuned-bhkrs","openshift-image-registry/node-ca-b6hff","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal","openshift-multus/network-metrics-daemon-vmvpp","openshift-network-diagnostics/network-check-target-b29qk","openshift-ovn-kubernetes/ovnkube-node-jvzv7"]
Sep 16 18:16:18.733492 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.733468    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.735874 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.735848    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:16:18.736009 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.735989    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fx5tq\""
Sep 16 18:16:18.736119 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.736098    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:16:18.736119 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.736114    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:16:18.736240 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.736221    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:16:18.736328 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.736309    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:18.736420 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:18.736393    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:18.739767 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.738730    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.741284 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.741263    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.741393 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.741368    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:16:18.743303 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.743283    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:16:18.743634 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.743498    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:16:18.743634 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.743515    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zlfxn\""
Sep 16 18:16:18.743814 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.743726    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:16:18.744228 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.744208    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:16:18.744449 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.744433    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:16:18.744663 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.744644    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-twspf\""
Sep 16 18:16:18.744901 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.744885    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:16:18.745133 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.745116    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:16:18.745786 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.745740    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:16:18.746529 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.745940    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:16:18.746529 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.745959    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dqj4w"
Sep 16 18:16:18.747720 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.747701    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:16:18.747852 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.747836    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.748683 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.748462    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:16:18.748683 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.748564    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kmk6v\""
Sep 16 18:16:18.750177 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.750155    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qhgwc\""
Sep 16 18:16:18.750448 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.750428    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:16:18.750547 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.750431    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:16:18.750946 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.750926    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.753545 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.753360    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:16:18.754178 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.754146    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:18.754289 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:18.754244    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:18.754358 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.754307    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:16:18.755458 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.755435    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-frzft\""
Sep 16 18:16:18.755594 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.755568    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-cnibin\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.755666 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.755621    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-run-systemd\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.755772 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.755751    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-run-openvswitch\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.755872 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.755785    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-run-ovn\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.755872 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.755816    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-slash\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.755872 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.755858    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.756039 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.755903    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-etc-selinux\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.756138 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756110    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-run-k8s-cni-cncf-io\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.756203 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756156    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-var-lib-cni-multus\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.756203 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756184    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-etc-openvswitch\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.756282 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756232    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d52c2148-fdca-4bb1-9ef7-b2492c8b7b57-hosts-file\") pod \"node-resolver-dqj4w\" (UID: \"d52c2148-fdca-4bb1-9ef7-b2492c8b7b57\") " pod="openshift-dns/node-resolver-dqj4w"
Sep 16 18:16:18.756344 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756306    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-run-multus-certs\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.756383 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756349    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsdsr\" (UniqueName: \"kubernetes.io/projected/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-kube-api-access-hsdsr\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.756433 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756391    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4665c154-7d62-47da-b6cb-7f221aa36343-cni-binary-copy\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.756488 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756436    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4665c154-7d62-47da-b6cb-7f221aa36343-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.756541 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756494    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3377376-cb87-41f0-afa7-edb4bd1b0350-multus-daemon-config\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.756596 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756556    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-run-netns\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.756636 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756599    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d52c2148-fdca-4bb1-9ef7-b2492c8b7b57-tmp-dir\") pod \"node-resolver-dqj4w\" (UID: \"d52c2148-fdca-4bb1-9ef7-b2492c8b7b57\") " pod="openshift-dns/node-resolver-dqj4w"
Sep 16 18:16:18.756671 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756635    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz6bb\" (UniqueName: \"kubernetes.io/projected/4665c154-7d62-47da-b6cb-7f221aa36343-kube-api-access-kz6bb\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.756706 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756671    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggngr\" (UniqueName: \"kubernetes.io/projected/b3377376-cb87-41f0-afa7-edb4bd1b0350-kube-api-access-ggngr\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.756737 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756713    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-systemd-units\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.756770 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756747    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kht8\" (UniqueName: \"kubernetes.io/projected/1bcb5cce-815d-4661-9e03-1aec2fbce49c-kube-api-access-9kht8\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.756808 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756782    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-device-dir\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.756846 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756813    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-sys-fs\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.756887 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756856    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4665c154-7d62-47da-b6cb-7f221aa36343-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.756940 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756900    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-log-socket\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.756940 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756929    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-run-ovn-kubernetes\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.757171 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756963    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-cni-bin\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.757171 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.756996    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-var-lib-kubelet\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.757171 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757029    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.757171 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757070    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4665c154-7d62-47da-b6cb-7f221aa36343-system-cni-dir\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.757171 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757122    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-var-lib-openvswitch\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.757171 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757162    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-env-overrides\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.757460 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757195    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-socket-dir\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.757460 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757266    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-multus-cni-dir\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.757460 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757302    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-hostroot\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.757460 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757352    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-multus-conf-dir\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.757460 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757382    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-kubelet\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.757460 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757433    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4665c154-7d62-47da-b6cb-7f221aa36343-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.757799 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757470    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-system-cni-dir\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.757799 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757508    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-run-netns\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.757799 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757563    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-ovnkube-config\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.757799 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757603    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-var-lib-cni-bin\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.757799 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757646    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-etc-kubernetes\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.757799 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757683    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-cni-netd\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.757799 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757719    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-registration-dir\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.757799 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757768    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wx9c\" (UniqueName: \"kubernetes.io/projected/d52c2148-fdca-4bb1-9ef7-b2492c8b7b57-kube-api-access-9wx9c\") pod \"node-resolver-dqj4w\" (UID: \"d52c2148-fdca-4bb1-9ef7-b2492c8b7b57\") " pod="openshift-dns/node-resolver-dqj4w"
Sep 16 18:16:18.758255 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757810    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-multus-socket-dir-parent\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.758255 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757835    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnc6b\" (UniqueName: \"kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b\") pod \"network-check-target-b29qk\" (UID: \"84e3cf97-48f5-415e-a5f5-a4cf55bda177\") " pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:18.758255 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757865    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4665c154-7d62-47da-b6cb-7f221aa36343-cnibin\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.758255 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757907    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3377376-cb87-41f0-afa7-edb4bd1b0350-cni-binary-copy\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.758255 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757932    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-node-log\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.758255 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.757972    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-ovn-node-metrics-cert\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.758255 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.758000    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-ovnkube-script-lib\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.759371 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.759073    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jtdvj"
Sep 16 18:16:18.761679 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.761635    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b6hff"
Sep 16 18:16:18.763576 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.763554    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:16:18.763670 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.763642    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dhcfw\""
Sep 16 18:16:18.763947 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.763931    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:16:18.764176 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.764155    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:18.764255 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:18.764223    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:18.764584 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.764540    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:16:18.764663 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.764583    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:16:18.764840 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.764819    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:16:18.765033 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.764987    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-85n7x\""
Sep 16 18:16:18.767094 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.766843    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cs2z7"
Sep 16 18:16:18.767094 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.766862    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:16:18.769322 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.768964    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:16:18.769322 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.769199    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:16:18.769322 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.769234    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:16:18.769520 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.769387    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d6tnz\""
Sep 16 18:16:18.788201 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.788176    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9f2xw"
Sep 16 18:16:18.796986 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.796964    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9f2xw"
Sep 16 18:16:18.847320 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.847184    2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:16:18.858520 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858486    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-socket-dir\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.858669 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858528    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-host\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.858669 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858553    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkvmj\" (UniqueName: \"kubernetes.io/projected/e1ee1605-7d05-4668-98e6-8249ac3e1052-kube-api-access-jkvmj\") pod \"node-ca-b6hff\" (UID: \"e1ee1605-7d05-4668-98e6-8249ac3e1052\") " pod="openshift-image-registry/node-ca-b6hff"
Sep 16 18:16:18.858669 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858591    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-multus-cni-dir\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.858669 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858617    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-hostroot\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.858669 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858639    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-multus-conf-dir\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.858669 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858664    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-kubelet\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858704    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-socket-dir\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858710    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4665c154-7d62-47da-b6cb-7f221aa36343-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858719    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-multus-conf-dir\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858723    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-hostroot\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858757    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-multus-cni-dir\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858783    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-system-cni-dir\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858809    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-run-netns\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858762    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-kubelet\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858835    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-ovnkube-config\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858846    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-system-cni-dir\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858865    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50b4667a-4d35-44cb-865b-747a72c7c3e6-dbus\") pod \"global-pull-secret-syncer-7w74g\" (UID: \"50b4667a-4d35-44cb-865b-747a72c7c3e6\") " pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858872    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-run-netns\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858899    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-var-lib-cni-bin\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858920    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-etc-kubernetes\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858935    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-cni-netd\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858969    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-registration-dir\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858988    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-var-lib-cni-bin\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.858998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.858994    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wx9c\" (UniqueName: \"kubernetes.io/projected/d52c2148-fdca-4bb1-9ef7-b2492c8b7b57-kube-api-access-9wx9c\") pod \"node-resolver-dqj4w\" (UID: \"d52c2148-fdca-4bb1-9ef7-b2492c8b7b57\") " pod="openshift-dns/node-resolver-dqj4w"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859038    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-sysconfig\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859064    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-kubernetes\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859087    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-systemd\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859119    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-multus-socket-dir-parent\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859143    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnc6b\" (UniqueName: \"kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b\") pod \"network-check-target-b29qk\" (UID: \"84e3cf97-48f5-415e-a5f5-a4cf55bda177\") " pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859169    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4665c154-7d62-47da-b6cb-7f221aa36343-cnibin\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859193    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-sysctl-conf\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859216    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-tmp\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859239    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs\") pod \"network-metrics-daemon-vmvpp\" (UID: \"e4c7d4e4-64cb-4cdd-90ae-db41d1889099\") " pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859279    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3377376-cb87-41f0-afa7-edb4bd1b0350-cni-binary-copy\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859322    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-node-log\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859319    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-etc-kubernetes\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859352    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-ovn-node-metrics-cert\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859357    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4665c154-7d62-47da-b6cb-7f221aa36343-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859382    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-cni-netd\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859407    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-ovnkube-config\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.859590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859438    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-ovnkube-script-lib\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859446    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-registration-dir\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859473    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-modprobe-d\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859495    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4665c154-7d62-47da-b6cb-7f221aa36343-cnibin\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859499    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-sys\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859803    2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859888    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-cnibin\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.859992    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-ovnkube-script-lib\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860194    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3377376-cb87-41f0-afa7-edb4bd1b0350-cni-binary-copy\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860201    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-node-log\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860217    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-run-systemd\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860227    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-multus-socket-dir-parent\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860254    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-cnibin\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860284    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-run-systemd\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860287    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-run-openvswitch\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860335    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-run-ovn\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860354    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-run-openvswitch\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.860386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860392    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50b4667a-4d35-44cb-865b-747a72c7c3e6-kubelet-config\") pod \"global-pull-secret-syncer-7w74g\" (UID: \"50b4667a-4d35-44cb-865b-747a72c7c3e6\") " pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860434    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-run-ovn\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860489    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2a707fc-feb5-4668-9207-cd0cf3c32fad-iptables-alerter-script\") pod \"iptables-alerter-cs2z7\" (UID: \"d2a707fc-feb5-4668-9207-cd0cf3c32fad\") " pod="openshift-network-operator/iptables-alerter-cs2z7"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860519    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-slash\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860545    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860567    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-etc-selinux\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860594    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-run\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860604    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-slash\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860618    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-tuned\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860625    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860646    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qflpx\" (UniqueName: \"kubernetes.io/projected/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-kube-api-access-qflpx\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860659    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-etc-selinux\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860676    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-run-k8s-cni-cncf-io\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860716    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-var-lib-cni-multus\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860713    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-run-k8s-cni-cncf-io\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860750    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-etc-openvswitch\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860766    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-var-lib-cni-multus\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.861129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860776    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d52c2148-fdca-4bb1-9ef7-b2492c8b7b57-hosts-file\") pod \"node-resolver-dqj4w\" (UID: \"d52c2148-fdca-4bb1-9ef7-b2492c8b7b57\") " pod="openshift-dns/node-resolver-dqj4w"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860801    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1ee1605-7d05-4668-98e6-8249ac3e1052-host\") pod \"node-ca-b6hff\" (UID: \"e1ee1605-7d05-4668-98e6-8249ac3e1052\") " pod="openshift-image-registry/node-ca-b6hff"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860809    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-etc-openvswitch\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860826    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2a707fc-feb5-4668-9207-cd0cf3c32fad-host-slash\") pod \"iptables-alerter-cs2z7\" (UID: \"d2a707fc-feb5-4668-9207-cd0cf3c32fad\") " pod="openshift-network-operator/iptables-alerter-cs2z7"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860853    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-run-multus-certs\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860860    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d52c2148-fdca-4bb1-9ef7-b2492c8b7b57-hosts-file\") pod \"node-resolver-dqj4w\" (UID: \"d52c2148-fdca-4bb1-9ef7-b2492c8b7b57\") " pod="openshift-dns/node-resolver-dqj4w"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860887    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsdsr\" (UniqueName: \"kubernetes.io/projected/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-kube-api-access-hsdsr\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860901    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-run-multus-certs\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860915    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4665c154-7d62-47da-b6cb-7f221aa36343-cni-binary-copy\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.860961    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4665c154-7d62-47da-b6cb-7f221aa36343-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861005    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-sysctl-d\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861028    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3377376-cb87-41f0-afa7-edb4bd1b0350-multus-daemon-config\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861079    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-run-netns\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861100    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d52c2148-fdca-4bb1-9ef7-b2492c8b7b57-tmp-dir\") pod \"node-resolver-dqj4w\" (UID: \"d52c2148-fdca-4bb1-9ef7-b2492c8b7b57\") " pod="openshift-dns/node-resolver-dqj4w"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861122    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz6bb\" (UniqueName: \"kubernetes.io/projected/4665c154-7d62-47da-b6cb-7f221aa36343-kube-api-access-kz6bb\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861152    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f06204d6-bd2b-45fe-b2bc-0814ad88beb7-agent-certs\") pod \"konnectivity-agent-jtdvj\" (UID: \"f06204d6-bd2b-45fe-b2bc-0814ad88beb7\") " pod="kube-system/konnectivity-agent-jtdvj"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861190    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggngr\" (UniqueName: \"kubernetes.io/projected/b3377376-cb87-41f0-afa7-edb4bd1b0350-kube-api-access-ggngr\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.861660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861211    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-systemd-units\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861232    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kht8\" (UniqueName: \"kubernetes.io/projected/1bcb5cce-815d-4661-9e03-1aec2fbce49c-kube-api-access-9kht8\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861278    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-device-dir\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861345    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-sys-fs\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861370    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4665c154-7d62-47da-b6cb-7f221aa36343-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861387    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4665c154-7d62-47da-b6cb-7f221aa36343-cni-binary-copy\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861394    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4k2h\" (UniqueName: \"kubernetes.io/projected/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-kube-api-access-t4k2h\") pod \"network-metrics-daemon-vmvpp\" (UID: \"e4c7d4e4-64cb-4cdd-90ae-db41d1889099\") " pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861466    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-device-dir\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861511    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-sys-fs\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861846    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3377376-cb87-41f0-afa7-edb4bd1b0350-multus-daemon-config\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.861906    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-run-netns\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862095    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-systemd-units\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862131    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1ee1605-7d05-4668-98e6-8249ac3e1052-serviceca\") pod \"node-ca-b6hff\" (UID: \"e1ee1605-7d05-4668-98e6-8249ac3e1052\") " pod="openshift-image-registry/node-ca-b6hff"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862161    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-log-socket\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862190    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d52c2148-fdca-4bb1-9ef7-b2492c8b7b57-tmp-dir\") pod \"node-resolver-dqj4w\" (UID: \"d52c2148-fdca-4bb1-9ef7-b2492c8b7b57\") " pod="openshift-dns/node-resolver-dqj4w"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862207    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-run-ovn-kubernetes\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862232    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-cni-bin\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.862454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862255    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-lib-modules\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862282    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-var-lib-kubelet\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862355    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-var-lib-kubelet\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862378    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4665c154-7d62-47da-b6cb-7f221aa36343-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862430    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-run-ovn-kubernetes\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862443    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862455    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-log-socket\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862383    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bcb5cce-815d-4661-9e03-1aec2fbce49c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862501    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-host-cni-bin\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862505    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3377376-cb87-41f0-afa7-edb4bd1b0350-host-var-lib-kubelet\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862538    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4665c154-7d62-47da-b6cb-7f221aa36343-system-cni-dir\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862569    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f06204d6-bd2b-45fe-b2bc-0814ad88beb7-konnectivity-ca\") pod \"konnectivity-agent-jtdvj\" (UID: \"f06204d6-bd2b-45fe-b2bc-0814ad88beb7\") " pod="kube-system/konnectivity-agent-jtdvj"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862587    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4665c154-7d62-47da-b6cb-7f221aa36343-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862592    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret\") pod \"global-pull-secret-syncer-7w74g\" (UID: \"50b4667a-4d35-44cb-865b-747a72c7c3e6\") " pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862627    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4665c154-7d62-47da-b6cb-7f221aa36343-system-cni-dir\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862631    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-var-lib-openvswitch\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862656    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-env-overrides\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.863334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862668    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-var-lib-openvswitch\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.864231 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.862701    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krrzq\" (UniqueName: \"kubernetes.io/projected/d2a707fc-feb5-4668-9207-cd0cf3c32fad-kube-api-access-krrzq\") pod \"iptables-alerter-cs2z7\" (UID: \"d2a707fc-feb5-4668-9207-cd0cf3c32fad\") " pod="openshift-network-operator/iptables-alerter-cs2z7"
Sep 16 18:16:18.864231 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.863051    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-env-overrides\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.864955 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.864931    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-ovn-node-metrics-cert\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.866051 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:18.866031    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:16:18.866147 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:18.866056    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:16:18.866147 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:18.866070    2554 projected.go:196] Error preparing data for projected volume kube-api-access-mnc6b for pod openshift-network-diagnostics/network-check-target-b29qk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:18.866378 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:18.866361    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b podName:84e3cf97-48f5-415e-a5f5-a4cf55bda177 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:19.366322762 +0000 UTC m=+2.925490599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mnc6b" (UniqueName: "kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b") pod "network-check-target-b29qk" (UID: "84e3cf97-48f5-415e-a5f5-a4cf55bda177") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:18.872154 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.872124    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggngr\" (UniqueName: \"kubernetes.io/projected/b3377376-cb87-41f0-afa7-edb4bd1b0350-kube-api-access-ggngr\") pod \"multus-blplt\" (UID: \"b3377376-cb87-41f0-afa7-edb4bd1b0350\") " pod="openshift-multus/multus-blplt"
Sep 16 18:16:18.872557 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.872532    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wx9c\" (UniqueName: \"kubernetes.io/projected/d52c2148-fdca-4bb1-9ef7-b2492c8b7b57-kube-api-access-9wx9c\") pod \"node-resolver-dqj4w\" (UID: \"d52c2148-fdca-4bb1-9ef7-b2492c8b7b57\") " pod="openshift-dns/node-resolver-dqj4w"
Sep 16 18:16:18.872644 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.872581    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kht8\" (UniqueName: \"kubernetes.io/projected/1bcb5cce-815d-4661-9e03-1aec2fbce49c-kube-api-access-9kht8\") pod \"aws-ebs-csi-driver-node-xjklz\" (UID: \"1bcb5cce-815d-4661-9e03-1aec2fbce49c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:18.873077 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.873054    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz6bb\" (UniqueName: \"kubernetes.io/projected/4665c154-7d62-47da-b6cb-7f221aa36343-kube-api-access-kz6bb\") pod \"multus-additional-cni-plugins-v4q6m\" (UID: \"4665c154-7d62-47da-b6cb-7f221aa36343\") " pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:18.873141 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.873106    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsdsr\" (UniqueName: \"kubernetes.io/projected/3b38bf5a-b91f-417e-8c74-14d351b0ba5f-kube-api-access-hsdsr\") pod \"ovnkube-node-jvzv7\" (UID: \"3b38bf5a-b91f-417e-8c74-14d351b0ba5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:18.882598 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.882550    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal" event={"ID":"5e066247b444270d650924fbf370e196","Type":"ContainerStarted","Data":"ed60c272a2009f4e206c8d57ec454bdd3da26561b4a6d02e7456a044210b1e6a"}
Sep 16 18:16:18.883750 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.883729    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal" event={"ID":"0f74f6b140c4aa2324e29421d89be3fe","Type":"ContainerStarted","Data":"cc8df6e03a4917c31a1a3ffa6298fdd2ff6c0c7d10790fa065ebbacd1f5c8050"}
Sep 16 18:16:18.963695 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.963654    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-sysconfig\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.963695 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.963699    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-kubernetes\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.963919 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.963727    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-systemd\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.963919 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.963748    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-sysconfig\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.963919 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.963766    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-sysctl-conf\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.963919 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.963795    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-tmp\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.963919 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.963819    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-kubernetes\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.963919 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.963845    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-systemd\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.963919 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.963814    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs\") pod \"network-metrics-daemon-vmvpp\" (UID: \"e4c7d4e4-64cb-4cdd-90ae-db41d1889099\") " pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:18.963919 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.963915    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-modprobe-d\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.963919 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:18.963920    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.963943    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-sys\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.963972    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-sysctl-conf\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.963974    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50b4667a-4d35-44cb-865b-747a72c7c3e6-kubelet-config\") pod \"global-pull-secret-syncer-7w74g\" (UID: \"50b4667a-4d35-44cb-865b-747a72c7c3e6\") " pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:18.963988    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs podName:e4c7d4e4-64cb-4cdd-90ae-db41d1889099 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:19.463969221 +0000 UTC m=+3.023137052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs") pod "network-metrics-daemon-vmvpp" (UID: "e4c7d4e4-64cb-4cdd-90ae-db41d1889099") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964022    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2a707fc-feb5-4668-9207-cd0cf3c32fad-iptables-alerter-script\") pod \"iptables-alerter-cs2z7\" (UID: \"d2a707fc-feb5-4668-9207-cd0cf3c32fad\") " pod="openshift-network-operator/iptables-alerter-cs2z7"
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964056    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-run\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964027    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-sys\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964081    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-tuned\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964119    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-modprobe-d\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964125    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-run\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964148    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qflpx\" (UniqueName: \"kubernetes.io/projected/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-kube-api-access-qflpx\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964194    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1ee1605-7d05-4668-98e6-8249ac3e1052-host\") pod \"node-ca-b6hff\" (UID: \"e1ee1605-7d05-4668-98e6-8249ac3e1052\") " pod="openshift-image-registry/node-ca-b6hff"
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964220    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2a707fc-feb5-4668-9207-cd0cf3c32fad-host-slash\") pod \"iptables-alerter-cs2z7\" (UID: \"d2a707fc-feb5-4668-9207-cd0cf3c32fad\") " pod="openshift-network-operator/iptables-alerter-cs2z7"
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964054    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/50b4667a-4d35-44cb-865b-747a72c7c3e6-kubelet-config\") pod \"global-pull-secret-syncer-7w74g\" (UID: \"50b4667a-4d35-44cb-865b-747a72c7c3e6\") " pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964249    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-sysctl-d\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964291    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2a707fc-feb5-4668-9207-cd0cf3c32fad-host-slash\") pod \"iptables-alerter-cs2z7\" (UID: \"d2a707fc-feb5-4668-9207-cd0cf3c32fad\") " pod="openshift-network-operator/iptables-alerter-cs2z7"
Sep 16 18:16:18.964413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964299    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f06204d6-bd2b-45fe-b2bc-0814ad88beb7-agent-certs\") pod \"konnectivity-agent-jtdvj\" (UID: \"f06204d6-bd2b-45fe-b2bc-0814ad88beb7\") " pod="kube-system/konnectivity-agent-jtdvj"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964334    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1ee1605-7d05-4668-98e6-8249ac3e1052-host\") pod \"node-ca-b6hff\" (UID: \"e1ee1605-7d05-4668-98e6-8249ac3e1052\") " pod="openshift-image-registry/node-ca-b6hff"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964334    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4k2h\" (UniqueName: \"kubernetes.io/projected/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-kube-api-access-t4k2h\") pod \"network-metrics-daemon-vmvpp\" (UID: \"e4c7d4e4-64cb-4cdd-90ae-db41d1889099\") " pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964370    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1ee1605-7d05-4668-98e6-8249ac3e1052-serviceca\") pod \"node-ca-b6hff\" (UID: \"e1ee1605-7d05-4668-98e6-8249ac3e1052\") " pod="openshift-image-registry/node-ca-b6hff"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964398    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-lib-modules\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964449    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-var-lib-kubelet\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964478    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f06204d6-bd2b-45fe-b2bc-0814ad88beb7-konnectivity-ca\") pod \"konnectivity-agent-jtdvj\" (UID: \"f06204d6-bd2b-45fe-b2bc-0814ad88beb7\") " pod="kube-system/konnectivity-agent-jtdvj"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964514    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret\") pod \"global-pull-secret-syncer-7w74g\" (UID: \"50b4667a-4d35-44cb-865b-747a72c7c3e6\") " pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964542    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krrzq\" (UniqueName: \"kubernetes.io/projected/d2a707fc-feb5-4668-9207-cd0cf3c32fad-kube-api-access-krrzq\") pod \"iptables-alerter-cs2z7\" (UID: \"d2a707fc-feb5-4668-9207-cd0cf3c32fad\") " pod="openshift-network-operator/iptables-alerter-cs2z7"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964565    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-host\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964593    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkvmj\" (UniqueName: \"kubernetes.io/projected/e1ee1605-7d05-4668-98e6-8249ac3e1052-kube-api-access-jkvmj\") pod \"node-ca-b6hff\" (UID: \"e1ee1605-7d05-4668-98e6-8249ac3e1052\") " pod="openshift-image-registry/node-ca-b6hff"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964622    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2a707fc-feb5-4668-9207-cd0cf3c32fad-iptables-alerter-script\") pod \"iptables-alerter-cs2z7\" (UID: \"d2a707fc-feb5-4668-9207-cd0cf3c32fad\") " pod="openshift-network-operator/iptables-alerter-cs2z7"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964635    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-sysctl-d\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964626    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50b4667a-4d35-44cb-865b-747a72c7c3e6-dbus\") pod \"global-pull-secret-syncer-7w74g\" (UID: \"50b4667a-4d35-44cb-865b-747a72c7c3e6\") " pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964686    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-var-lib-kubelet\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964744    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/50b4667a-4d35-44cb-865b-747a72c7c3e6-dbus\") pod \"global-pull-secret-syncer-7w74g\" (UID: \"50b4667a-4d35-44cb-865b-747a72c7c3e6\") " pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964775    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-lib-modules\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964794    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-host\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.965280 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:18.964866    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:18.966214 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.964878    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1ee1605-7d05-4668-98e6-8249ac3e1052-serviceca\") pod \"node-ca-b6hff\" (UID: \"e1ee1605-7d05-4668-98e6-8249ac3e1052\") " pod="openshift-image-registry/node-ca-b6hff"
Sep 16 18:16:18.966214 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:18.964915    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret podName:50b4667a-4d35-44cb-865b-747a72c7c3e6 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:19.464898255 +0000 UTC m=+3.024066075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret") pod "global-pull-secret-syncer-7w74g" (UID: "50b4667a-4d35-44cb-865b-747a72c7c3e6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:18.966214 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.965293    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f06204d6-bd2b-45fe-b2bc-0814ad88beb7-konnectivity-ca\") pod \"konnectivity-agent-jtdvj\" (UID: \"f06204d6-bd2b-45fe-b2bc-0814ad88beb7\") " pod="kube-system/konnectivity-agent-jtdvj"
Sep 16 18:16:18.967019 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.966992    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-etc-tuned\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.967144 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.967000    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-tmp\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.967520 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.967499    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f06204d6-bd2b-45fe-b2bc-0814ad88beb7-agent-certs\") pod \"konnectivity-agent-jtdvj\" (UID: \"f06204d6-bd2b-45fe-b2bc-0814ad88beb7\") " pod="kube-system/konnectivity-agent-jtdvj"
Sep 16 18:16:18.976034 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.975965    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qflpx\" (UniqueName: \"kubernetes.io/projected/005fa8ea-ab42-45e2-ab57-6a1cc067a32b-kube-api-access-qflpx\") pod \"tuned-bhkrs\" (UID: \"005fa8ea-ab42-45e2-ab57-6a1cc067a32b\") " pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:18.977353 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.977327    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4k2h\" (UniqueName: \"kubernetes.io/projected/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-kube-api-access-t4k2h\") pod \"network-metrics-daemon-vmvpp\" (UID: \"e4c7d4e4-64cb-4cdd-90ae-db41d1889099\") " pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:18.977500 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.977466    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkvmj\" (UniqueName: \"kubernetes.io/projected/e1ee1605-7d05-4668-98e6-8249ac3e1052-kube-api-access-jkvmj\") pod \"node-ca-b6hff\" (UID: \"e1ee1605-7d05-4668-98e6-8249ac3e1052\") " pod="openshift-image-registry/node-ca-b6hff"
Sep 16 18:16:18.977805 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:18.977781    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krrzq\" (UniqueName: \"kubernetes.io/projected/d2a707fc-feb5-4668-9207-cd0cf3c32fad-kube-api-access-krrzq\") pod \"iptables-alerter-cs2z7\" (UID: \"d2a707fc-feb5-4668-9207-cd0cf3c32fad\") " pod="openshift-network-operator/iptables-alerter-cs2z7"
Sep 16 18:16:19.049064 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:19.049024    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-blplt"
Sep 16 18:16:19.058703 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:19.058677    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:19.067849 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:19.067826    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz"
Sep 16 18:16:19.073530 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:19.073500    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dqj4w"
Sep 16 18:16:19.082384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:19.082365    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v4q6m"
Sep 16 18:16:19.090099 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:19.090079    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bhkrs"
Sep 16 18:16:19.099004 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:19.098987    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jtdvj"
Sep 16 18:16:19.107729 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:19.107705    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b6hff"
Sep 16 18:16:19.114457 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:19.114439    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cs2z7"
Sep 16 18:16:19.368822 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:19.368733    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnc6b\" (UniqueName: \"kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b\") pod \"network-check-target-b29qk\" (UID: \"84e3cf97-48f5-415e-a5f5-a4cf55bda177\") " pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:19.369012 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:19.368914    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:16:19.369012 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:19.368944    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:16:19.369012 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:19.368966    2554 projected.go:196] Error preparing data for projected volume kube-api-access-mnc6b for pod openshift-network-diagnostics/network-check-target-b29qk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:19.369185 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:19.369028    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b podName:84e3cf97-48f5-415e-a5f5-a4cf55bda177 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:20.369013192 +0000 UTC m=+3.928181021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnc6b" (UniqueName: "kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b") pod "network-check-target-b29qk" (UID: "84e3cf97-48f5-415e-a5f5-a4cf55bda177") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:19.469582 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:19.469536    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret\") pod \"global-pull-secret-syncer-7w74g\" (UID: \"50b4667a-4d35-44cb-865b-747a72c7c3e6\") " pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:19.469782 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:19.469626    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs\") pod \"network-metrics-daemon-vmvpp\" (UID: \"e4c7d4e4-64cb-4cdd-90ae-db41d1889099\") " pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:19.469782 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:19.469691    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:19.469782 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:19.469750    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:19.469782 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:19.469781    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret podName:50b4667a-4d35-44cb-865b-747a72c7c3e6 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:20.46975891 +0000 UTC m=+4.028926731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret") pod "global-pull-secret-syncer-7w74g" (UID: "50b4667a-4d35-44cb-865b-747a72c7c3e6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:19.470013 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:19.469806    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs podName:e4c7d4e4-64cb-4cdd-90ae-db41d1889099 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:20.469788848 +0000 UTC m=+4.028956677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs") pod "network-metrics-daemon-vmvpp" (UID: "e4c7d4e4-64cb-4cdd-90ae-db41d1889099") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:19.801410 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:19.801329    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:11:19 +0000 UTC" deadline="2028-05-16 08:39:12.330847899 +0000 UTC"
Sep 16 18:16:19.801410 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:19.801360    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14582h22m52.529491332s"
Sep 16 18:16:19.877951 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:19.877920    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:19.878102 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:19.878026    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:19.947557 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:19.947533    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4665c154_7d62_47da_b6cb_7f221aa36343.slice/crio-f626b58e31b1810b265f97b731422db3f4889070765291cfea0da85604ce819a WatchSource:0}: Error finding container f626b58e31b1810b265f97b731422db3f4889070765291cfea0da85604ce819a: Status 404 returned error can't find the container with id f626b58e31b1810b265f97b731422db3f4889070765291cfea0da85604ce819a
Sep 16 18:16:19.948868 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:19.948833    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3377376_cb87_41f0_afa7_edb4bd1b0350.slice/crio-e22b6b84c3c15750ce35bae40877f45afb0441569bb8450e6f952ae0667521fc WatchSource:0}: Error finding container e22b6b84c3c15750ce35bae40877f45afb0441569bb8450e6f952ae0667521fc: Status 404 returned error can't find the container with id e22b6b84c3c15750ce35bae40877f45afb0441569bb8450e6f952ae0667521fc
Sep 16 18:16:19.950393 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:19.950360    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52c2148_fdca_4bb1_9ef7_b2492c8b7b57.slice/crio-9ca5939dc882de7b120242b6dd405c9d7d494846dc7ee2e57d9a3890841501d4 WatchSource:0}: Error finding container 9ca5939dc882de7b120242b6dd405c9d7d494846dc7ee2e57d9a3890841501d4: Status 404 returned error can't find the container with id 9ca5939dc882de7b120242b6dd405c9d7d494846dc7ee2e57d9a3890841501d4
Sep 16 18:16:19.951975 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:19.951921    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06204d6_bd2b_45fe_b2bc_0814ad88beb7.slice/crio-ca37e020389fdd60a400789719a51198b126adc70592a8b0de4624503f4ce301 WatchSource:0}: Error finding container ca37e020389fdd60a400789719a51198b126adc70592a8b0de4624503f4ce301: Status 404 returned error can't find the container with id ca37e020389fdd60a400789719a51198b126adc70592a8b0de4624503f4ce301
Sep 16 18:16:19.976097 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:19.976072    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a707fc_feb5_4668_9207_cd0cf3c32fad.slice/crio-d65f172a3fb2cba293d91d91bd0f481f4a5d0b213413995d23d3d9be2f0b8889 WatchSource:0}: Error finding container d65f172a3fb2cba293d91d91bd0f481f4a5d0b213413995d23d3d9be2f0b8889: Status 404 returned error can't find the container with id d65f172a3fb2cba293d91d91bd0f481f4a5d0b213413995d23d3d9be2f0b8889
Sep 16 18:16:19.977136 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:19.977114    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005fa8ea_ab42_45e2_ab57_6a1cc067a32b.slice/crio-05552b071251f2c2fb8ffb9730a9114178898d22cac9e452c238a7fb9fac61f5 WatchSource:0}: Error finding container 05552b071251f2c2fb8ffb9730a9114178898d22cac9e452c238a7fb9fac61f5: Status 404 returned error can't find the container with id 05552b071251f2c2fb8ffb9730a9114178898d22cac9e452c238a7fb9fac61f5
Sep 16 18:16:19.978689 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:19.978664    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ee1605_7d05_4668_98e6_8249ac3e1052.slice/crio-a58f2781170b9e0b900bd5b6fbbe432a802ac550a3d20da1c2f82e2cd2869404 WatchSource:0}: Error finding container a58f2781170b9e0b900bd5b6fbbe432a802ac550a3d20da1c2f82e2cd2869404: Status 404 returned error can't find the container with id a58f2781170b9e0b900bd5b6fbbe432a802ac550a3d20da1c2f82e2cd2869404
Sep 16 18:16:19.979547 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:19.979527    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b38bf5a_b91f_417e_8c74_14d351b0ba5f.slice/crio-31494124af2039977f5675a9b2076918376a3759721ba34671290919f5178919 WatchSource:0}: Error finding container 31494124af2039977f5675a9b2076918376a3759721ba34671290919f5178919: Status 404 returned error can't find the container with id 31494124af2039977f5675a9b2076918376a3759721ba34671290919f5178919
Sep 16 18:16:20.376861 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.376540    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnc6b\" (UniqueName: \"kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b\") pod \"network-check-target-b29qk\" (UID: \"84e3cf97-48f5-415e-a5f5-a4cf55bda177\") " pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:20.376861 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:20.376759    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:16:20.377051 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:20.376894    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:16:20.377051 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:20.376912    2554 projected.go:196] Error preparing data for projected volume kube-api-access-mnc6b for pod openshift-network-diagnostics/network-check-target-b29qk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:20.377051 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:20.376967    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b podName:84e3cf97-48f5-415e-a5f5-a4cf55bda177 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:22.376953139 +0000 UTC m=+5.936120967 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnc6b" (UniqueName: "kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b") pod "network-check-target-b29qk" (UID: "84e3cf97-48f5-415e-a5f5-a4cf55bda177") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:20.478080 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.477317    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret\") pod \"global-pull-secret-syncer-7w74g\" (UID: \"50b4667a-4d35-44cb-865b-747a72c7c3e6\") " pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:20.478080 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.477396    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs\") pod \"network-metrics-daemon-vmvpp\" (UID: \"e4c7d4e4-64cb-4cdd-90ae-db41d1889099\") " pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:20.478080 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:20.477537    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:20.478080 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:20.477601    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs podName:e4c7d4e4-64cb-4cdd-90ae-db41d1889099 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:22.477581395 +0000 UTC m=+6.036749225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs") pod "network-metrics-daemon-vmvpp" (UID: "e4c7d4e4-64cb-4cdd-90ae-db41d1889099") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:20.478080 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:20.477989    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:20.478080 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:20.478050    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret podName:50b4667a-4d35-44cb-865b-747a72c7c3e6 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:22.478034914 +0000 UTC m=+6.037202733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret") pod "global-pull-secret-syncer-7w74g" (UID: "50b4667a-4d35-44cb-865b-747a72c7c3e6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:20.802689 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.802484    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:11:19 +0000 UTC" deadline="2028-02-24 20:46:34.523773057 +0000 UTC"
Sep 16 18:16:20.802689 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.802568    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12626h30m13.721210379s"
Sep 16 18:16:20.879058 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.878441    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:20.879058 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.878449    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:20.879058 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:20.878579    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:20.879058 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:20.878673    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:20.901876 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.901783    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz" event={"ID":"1bcb5cce-815d-4661-9e03-1aec2fbce49c","Type":"ContainerStarted","Data":"cb1b54420c1a8b614a6d95d560ca7f575df504bec0b7864338455044aa5c1ebf"}
Sep 16 18:16:20.907843 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.907808    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7" event={"ID":"3b38bf5a-b91f-417e-8c74-14d351b0ba5f","Type":"ContainerStarted","Data":"31494124af2039977f5675a9b2076918376a3759721ba34671290919f5178919"}
Sep 16 18:16:20.914887 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.914821    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bhkrs" event={"ID":"005fa8ea-ab42-45e2-ab57-6a1cc067a32b","Type":"ContainerStarted","Data":"05552b071251f2c2fb8ffb9730a9114178898d22cac9e452c238a7fb9fac61f5"}
Sep 16 18:16:20.928062 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.927995    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jtdvj" event={"ID":"f06204d6-bd2b-45fe-b2bc-0814ad88beb7","Type":"ContainerStarted","Data":"ca37e020389fdd60a400789719a51198b126adc70592a8b0de4624503f4ce301"}
Sep 16 18:16:20.934016 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.933955    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dqj4w" event={"ID":"d52c2148-fdca-4bb1-9ef7-b2492c8b7b57","Type":"ContainerStarted","Data":"9ca5939dc882de7b120242b6dd405c9d7d494846dc7ee2e57d9a3890841501d4"}
Sep 16 18:16:20.944210 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.944182    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4q6m" event={"ID":"4665c154-7d62-47da-b6cb-7f221aa36343","Type":"ContainerStarted","Data":"f626b58e31b1810b265f97b731422db3f4889070765291cfea0da85604ce819a"}
Sep 16 18:16:20.949213 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.948442    2554 generic.go:356] "Generic (PLEG): container finished" podID="5e066247b444270d650924fbf370e196" containerID="16cfadcbd78c7ce25c9046391996a8c6042c9a3fdc7a9ae5f16f91050bcf83f7" exitCode=0
Sep 16 18:16:20.949213 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.948755    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal" event={"ID":"5e066247b444270d650924fbf370e196","Type":"ContainerDied","Data":"16cfadcbd78c7ce25c9046391996a8c6042c9a3fdc7a9ae5f16f91050bcf83f7"}
Sep 16 18:16:20.955704 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.955677    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b6hff" event={"ID":"e1ee1605-7d05-4668-98e6-8249ac3e1052","Type":"ContainerStarted","Data":"a58f2781170b9e0b900bd5b6fbbe432a802ac550a3d20da1c2f82e2cd2869404"}
Sep 16 18:16:20.990060 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:20.989994    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cs2z7" event={"ID":"d2a707fc-feb5-4668-9207-cd0cf3c32fad","Type":"ContainerStarted","Data":"d65f172a3fb2cba293d91d91bd0f481f4a5d0b213413995d23d3d9be2f0b8889"}
Sep 16 18:16:21.005714 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:21.005657    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-blplt" event={"ID":"b3377376-cb87-41f0-afa7-edb4bd1b0350","Type":"ContainerStarted","Data":"e22b6b84c3c15750ce35bae40877f45afb0441569bb8450e6f952ae0667521fc"}
Sep 16 18:16:21.024197 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:21.023072    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal" event={"ID":"0f74f6b140c4aa2324e29421d89be3fe","Type":"ContainerStarted","Data":"c1f5574b832ef44664d3ecc28d4d5db94feeaa7d3cd83767073dfb6ec389aac9"}
Sep 16 18:16:21.041188 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:21.041129    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal" podStartSLOduration=3.041107471 podStartE2EDuration="3.041107471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:21.040517723 +0000 UTC m=+4.599685580" watchObservedRunningTime="2026-09-16 18:16:21.041107471 +0000 UTC m=+4.600275310"
Sep 16 18:16:21.877835 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:21.877800    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:21.878494 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:21.877938    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:22.050744 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:22.050655    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal" event={"ID":"5e066247b444270d650924fbf370e196","Type":"ContainerStarted","Data":"9d30b5fc091627f54bc0026c22a218ae47dc70472e18ef1d5b522fe4ecfcc684"}
Sep 16 18:16:22.394626 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:22.394567    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnc6b\" (UniqueName: \"kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b\") pod \"network-check-target-b29qk\" (UID: \"84e3cf97-48f5-415e-a5f5-a4cf55bda177\") " pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:22.394831 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:22.394782    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:16:22.394831 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:22.394804    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:16:22.394831 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:22.394818    2554 projected.go:196] Error preparing data for projected volume kube-api-access-mnc6b for pod openshift-network-diagnostics/network-check-target-b29qk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:22.395010 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:22.394897    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b podName:84e3cf97-48f5-415e-a5f5-a4cf55bda177 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:26.394875265 +0000 UTC m=+9.954043083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnc6b" (UniqueName: "kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b") pod "network-check-target-b29qk" (UID: "84e3cf97-48f5-415e-a5f5-a4cf55bda177") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:22.495643 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:22.495599    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret\") pod \"global-pull-secret-syncer-7w74g\" (UID: \"50b4667a-4d35-44cb-865b-747a72c7c3e6\") " pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:22.495836 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:22.495692    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs\") pod \"network-metrics-daemon-vmvpp\" (UID: \"e4c7d4e4-64cb-4cdd-90ae-db41d1889099\") " pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:22.495836 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:22.495798    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:22.495836 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:22.495813    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:22.495981 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:22.495869    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs podName:e4c7d4e4-64cb-4cdd-90ae-db41d1889099 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:26.495849771 +0000 UTC m=+10.055017593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs") pod "network-metrics-daemon-vmvpp" (UID: "e4c7d4e4-64cb-4cdd-90ae-db41d1889099") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:22.495981 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:22.495889    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret podName:50b4667a-4d35-44cb-865b-747a72c7c3e6 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:26.495878488 +0000 UTC m=+10.055046302 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret") pod "global-pull-secret-syncer-7w74g" (UID: "50b4667a-4d35-44cb-865b-747a72c7c3e6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:22.878806 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:22.878142    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:22.878806 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:22.878284    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:22.878806 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:22.878142    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:22.878806 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:22.878422    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:23.877608 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:23.877571    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:23.877792 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:23.877711    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:24.879517 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:24.879480    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:24.880012 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:24.879698    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:24.880012 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:24.879826    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:24.880012 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:24.879694    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:25.877705 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:25.877499    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:25.877705 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:25.877653    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:26.431480 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:26.431435    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnc6b\" (UniqueName: \"kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b\") pod \"network-check-target-b29qk\" (UID: \"84e3cf97-48f5-415e-a5f5-a4cf55bda177\") " pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:26.432036 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:26.431687    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:16:26.432036 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:26.431711    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:16:26.432036 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:26.431725    2554 projected.go:196] Error preparing data for projected volume kube-api-access-mnc6b for pod openshift-network-diagnostics/network-check-target-b29qk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:26.432036 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:26.431792    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b podName:84e3cf97-48f5-415e-a5f5-a4cf55bda177 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:34.431772922 +0000 UTC m=+17.990940760 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnc6b" (UniqueName: "kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b") pod "network-check-target-b29qk" (UID: "84e3cf97-48f5-415e-a5f5-a4cf55bda177") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:26.532691 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:26.532654    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs\") pod \"network-metrics-daemon-vmvpp\" (UID: \"e4c7d4e4-64cb-4cdd-90ae-db41d1889099\") " pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:26.532885 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:26.532725    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret\") pod \"global-pull-secret-syncer-7w74g\" (UID: \"50b4667a-4d35-44cb-865b-747a72c7c3e6\") " pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:26.532885 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:26.532774    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:26.532885 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:26.532810    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:26.532885 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:26.532854    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret podName:50b4667a-4d35-44cb-865b-747a72c7c3e6 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:34.532835985 +0000 UTC m=+18.092003813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret") pod "global-pull-secret-syncer-7w74g" (UID: "50b4667a-4d35-44cb-865b-747a72c7c3e6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:26.532885 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:26.532872    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs podName:e4c7d4e4-64cb-4cdd-90ae-db41d1889099 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:34.532862855 +0000 UTC m=+18.092030669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs") pod "network-metrics-daemon-vmvpp" (UID: "e4c7d4e4-64cb-4cdd-90ae-db41d1889099") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:26.879536 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:26.879399    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:26.879536 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:26.879509    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:26.879772 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:26.879610    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:26.879772 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:26.879686    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:27.878522 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:27.878010    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:27.878522 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:27.878133    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:28.878407 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:28.878376    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:28.878572 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:28.878552    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:28.878932 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:28.878563    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:28.878932 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:28.878669    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:29.877830 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:29.877789    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:29.878020 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:29.877930    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:30.877442 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:30.877408    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:30.877874 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:30.877552    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:30.877874 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:30.877621    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:30.877874 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:30.877718    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:31.877629 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:31.877590    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:31.878064 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:31.877715    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:32.877500 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:32.877464    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:32.877663 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:32.877586    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:32.877663 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:32.877649    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:32.878066 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:32.877754    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:33.877666 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:33.877634    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:33.878167 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:33.877760    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:34.495245 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:34.495197    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnc6b\" (UniqueName: \"kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b\") pod \"network-check-target-b29qk\" (UID: \"84e3cf97-48f5-415e-a5f5-a4cf55bda177\") " pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:34.495429 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:34.495379    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:16:34.495429 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:34.495401    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:16:34.495429 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:34.495411    2554 projected.go:196] Error preparing data for projected volume kube-api-access-mnc6b for pod openshift-network-diagnostics/network-check-target-b29qk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:34.495588 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:34.495462    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b podName:84e3cf97-48f5-415e-a5f5-a4cf55bda177 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:50.495447937 +0000 UTC m=+34.054615749 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnc6b" (UniqueName: "kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b") pod "network-check-target-b29qk" (UID: "84e3cf97-48f5-415e-a5f5-a4cf55bda177") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:34.596378 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:34.596340    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs\") pod \"network-metrics-daemon-vmvpp\" (UID: \"e4c7d4e4-64cb-4cdd-90ae-db41d1889099\") " pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:34.596557 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:34.596422    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret\") pod \"global-pull-secret-syncer-7w74g\" (UID: \"50b4667a-4d35-44cb-865b-747a72c7c3e6\") " pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:34.596557 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:34.596456    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:34.596557 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:34.596510    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:34.596719 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:34.596524    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs podName:e4c7d4e4-64cb-4cdd-90ae-db41d1889099 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:50.596505655 +0000 UTC m=+34.155673470 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs") pod "network-metrics-daemon-vmvpp" (UID: "e4c7d4e4-64cb-4cdd-90ae-db41d1889099") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:34.596719 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:34.596591    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret podName:50b4667a-4d35-44cb-865b-747a72c7c3e6 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:50.596581965 +0000 UTC m=+34.155749777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret") pod "global-pull-secret-syncer-7w74g" (UID: "50b4667a-4d35-44cb-865b-747a72c7c3e6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:34.877657 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:34.877593    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:34.877783 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:34.877693    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:34.878053 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:34.877775    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:34.878053 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:34.877871    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:35.877556 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:35.877524    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:35.877702 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:35.877646    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:36.878794 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:36.878587    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:36.879319 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:36.878700    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:36.879319 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:36.878895    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:36.879319 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:36.878950    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:37.877418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:37.877383    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:37.877602 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:37.877503    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:38.878001 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:38.877836    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:38.878579 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:38.877904    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:38.878579 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:38.878102    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:38.878579 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:38.878177    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:39.087109 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.087075    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b6hff" event={"ID":"e1ee1605-7d05-4668-98e6-8249ac3e1052","Type":"ContainerStarted","Data":"ef8bee1d3a6ed3467c80f911fdc7ea6256c08be6f633266d9a8ccec10df66515"}
Sep 16 18:16:39.088658 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.088631    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-blplt" event={"ID":"b3377376-cb87-41f0-afa7-edb4bd1b0350","Type":"ContainerStarted","Data":"c8c868a18840f5e419cf1bc2683ca555ccc61544ef7ec26f8e4e4330b5951637"}
Sep 16 18:16:39.090041 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.090019    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz" event={"ID":"1bcb5cce-815d-4661-9e03-1aec2fbce49c","Type":"ContainerStarted","Data":"eaa77fe3a4fb59df577b7e77c9e3d9ca48fd21d3635d61038d9ff7ff298ea007"}
Sep 16 18:16:39.092591 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.092568    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7" event={"ID":"3b38bf5a-b91f-417e-8c74-14d351b0ba5f","Type":"ContainerStarted","Data":"59c041a564523820651a85a4c714492da828619eb5ccd8890cc3d0582b080d3c"}
Sep 16 18:16:39.092678 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.092631    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7" event={"ID":"3b38bf5a-b91f-417e-8c74-14d351b0ba5f","Type":"ContainerStarted","Data":"2340223cfa961ca3ee5d12c480fcfb4ef84221e30d4fe48e7e6c2238aafc6550"}
Sep 16 18:16:39.092678 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.092661    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7" event={"ID":"3b38bf5a-b91f-417e-8c74-14d351b0ba5f","Type":"ContainerStarted","Data":"e5016d2dcceed75c2722225f9b71b6742406830c2b5ec9fa0e7c25b4eb20ac9e"}
Sep 16 18:16:39.092793 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.092696    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7" event={"ID":"3b38bf5a-b91f-417e-8c74-14d351b0ba5f","Type":"ContainerStarted","Data":"8b3b03c3f39c26740d953d0932add0630ea19817b28e00c9d5e5f246c220fc30"}
Sep 16 18:16:39.092793 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.092726    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7" event={"ID":"3b38bf5a-b91f-417e-8c74-14d351b0ba5f","Type":"ContainerStarted","Data":"4402a4a952fb4a555e3b1520204f8f624ae47292bace91031c59d8ddfcff94e8"}
Sep 16 18:16:39.093912 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.093894    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bhkrs" event={"ID":"005fa8ea-ab42-45e2-ab57-6a1cc067a32b","Type":"ContainerStarted","Data":"15c0dcf3234034bde1a1ef354f791b8300bc5f53f66f2b165c200346a46724da"}
Sep 16 18:16:39.095254 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.095216    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jtdvj" event={"ID":"f06204d6-bd2b-45fe-b2bc-0814ad88beb7","Type":"ContainerStarted","Data":"6a54b90c08ee44bb2b44d773bd393931c37dc390610b1dfbcda9525d244b7dbf"}
Sep 16 18:16:39.096551 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.096523    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dqj4w" event={"ID":"d52c2148-fdca-4bb1-9ef7-b2492c8b7b57","Type":"ContainerStarted","Data":"7050c08ae9b45143656d32d3c16248a39071c8359e768edc98245131abb3cdf3"}
Sep 16 18:16:39.098117 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.098085    2554 generic.go:356] "Generic (PLEG): container finished" podID="4665c154-7d62-47da-b6cb-7f221aa36343" containerID="9126263476e816754d870f988b403a6a9d25e3c42fb607596402ee5e85055dde" exitCode=0
Sep 16 18:16:39.098210 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.098118    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4q6m" event={"ID":"4665c154-7d62-47da-b6cb-7f221aa36343","Type":"ContainerDied","Data":"9126263476e816754d870f988b403a6a9d25e3c42fb607596402ee5e85055dde"}
Sep 16 18:16:39.105763 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.105722    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-b6hff" podStartSLOduration=3.67087211 podStartE2EDuration="22.105710823s" totalImagesPullingTime="18.434838713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:39.10528072 +0000 UTC m=+22.664448554" watchObservedRunningTime="2026-09-16 18:16:39.105710823 +0000 UTC m=+22.664878657"
Sep 16 18:16:39.105852 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.105788    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal" podStartSLOduration=21.105782774 podStartE2EDuration="21.105782774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:22.071670852 +0000 UTC m=+5.630838691" watchObservedRunningTime="2026-09-16 18:16:39.105782774 +0000 UTC m=+22.664950608"
Sep 16 18:16:39.165343 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.165283    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bhkrs" podStartSLOduration=3.722822406 podStartE2EDuration="22.165261321s" totalImagesPullingTime="18.442438915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:39.164998452 +0000 UTC m=+22.724166286" watchObservedRunningTime="2026-09-16 18:16:39.165261321 +0000 UTC m=+22.724429155"
Sep 16 18:16:39.188088 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.187985    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jtdvj" podStartSLOduration=3.7412090940000002 podStartE2EDuration="22.187963177s" totalImagesPullingTime="18.446754083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:39.187611025 +0000 UTC m=+22.746778863" watchObservedRunningTime="2026-09-16 18:16:39.187963177 +0000 UTC m=+22.747131014"
Sep 16 18:16:39.206851 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.206793    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-blplt" podStartSLOduration=3.677359054 podStartE2EDuration="22.206771844s" totalImagesPullingTime="18.52941279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:39.206362403 +0000 UTC m=+22.765530236" watchObservedRunningTime="2026-09-16 18:16:39.206771844 +0000 UTC m=+22.765939681"
Sep 16 18:16:39.222684 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.222629    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dqj4w" podStartSLOduration=3.769324444 podStartE2EDuration="22.222616621s" totalImagesPullingTime="18.453292177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:39.222210252 +0000 UTC m=+22.781378087" watchObservedRunningTime="2026-09-16 18:16:39.222616621 +0000 UTC m=+22.781784455"
Sep 16 18:16:39.877286 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.877257    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:39.877460 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:39.877377    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:39.980609 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:39.980574    2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:16:40.102387 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:40.102291    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cs2z7" event={"ID":"d2a707fc-feb5-4668-9207-cd0cf3c32fad","Type":"ContainerStarted","Data":"f17a42133bed3400b91104a6308b718e77917c8ad44dd9a6185f2c00cfaf451f"}
Sep 16 18:16:40.104193 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:40.104162    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz" event={"ID":"1bcb5cce-815d-4661-9e03-1aec2fbce49c","Type":"ContainerStarted","Data":"81317c97cf9c47e26ae495b46e3ef02ea3fb840a416e5be5cafd36f55d434dd3"}
Sep 16 18:16:40.107886 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:40.107854    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7" event={"ID":"3b38bf5a-b91f-417e-8c74-14d351b0ba5f","Type":"ContainerStarted","Data":"31bc2c9c73ad5916d4136e06af2f8172d01819029330a76688d6a60161e71346"}
Sep 16 18:16:40.121435 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:40.121390    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cs2z7" podStartSLOduration=3.680887008 podStartE2EDuration="22.121376478s" totalImagesPullingTime="18.44048947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:40.120664593 +0000 UTC m=+23.679832427" watchObservedRunningTime="2026-09-16 18:16:40.121376478 +0000 UTC m=+23.680544311"
Sep 16 18:16:40.805412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:40.805291    2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:16:39.980593401Z","UUID":"173b1840-3bbc-4068-a720-218224e849fd","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:16:40.807001 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:40.806970    2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:16:40.807126 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:40.807007    2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:16:40.880461 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:40.880435    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:40.880625 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:40.880560    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:40.880625 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:40.880589    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:40.880742 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:40.880705    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:41.877556 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:41.877347    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:41.877940 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:41.877660    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:42.114189 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:42.114131    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz" event={"ID":"1bcb5cce-815d-4661-9e03-1aec2fbce49c","Type":"ContainerStarted","Data":"6402ab316690235833d092feff2250edc4ad32aeeba015a515a962ab667225d7"}
Sep 16 18:16:42.117856 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:42.117832    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7" event={"ID":"3b38bf5a-b91f-417e-8c74-14d351b0ba5f","Type":"ContainerStarted","Data":"1dfb6eee4b139236f56d6c43b25a44bc76ff134060af72ffcbc50f1bf6e4312f"}
Sep 16 18:16:42.124954 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:42.124917    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jtdvj"
Sep 16 18:16:42.128675 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:42.125912    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jtdvj"
Sep 16 18:16:42.149536 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:42.149494    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xjklz" podStartSLOduration=4.540700102 podStartE2EDuration="25.149479818s" totalImagesPullingTime="20.608779716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:42.134423374 +0000 UTC m=+25.693591210" watchObservedRunningTime="2026-09-16 18:16:42.149479818 +0000 UTC m=+25.708647652"
Sep 16 18:16:42.881249 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:42.881208    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:42.881686 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:42.881347    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:42.881686 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:42.881388    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:42.881686 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:42.881471    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:43.120924 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:43.120888    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jtdvj"
Sep 16 18:16:43.121269 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:43.121249    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jtdvj"
Sep 16 18:16:43.878326 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:43.878120    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:43.878467 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:43.878351    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:44.126144 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:44.126112    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7" event={"ID":"3b38bf5a-b91f-417e-8c74-14d351b0ba5f","Type":"ContainerStarted","Data":"2474eb534028e4420e9800c52c3e86a6b1a18c82aef81d157ff666cd368eec3a"}
Sep 16 18:16:44.126669 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:44.126275    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:44.126669 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:44.126354    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:44.128137 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:44.128114    2554 generic.go:356] "Generic (PLEG): container finished" podID="4665c154-7d62-47da-b6cb-7f221aa36343" containerID="b5154b3afa7c45a1bcc83d09599f49cda2f9c3d188a0fa89c152fe78645796c9" exitCode=0
Sep 16 18:16:44.128260 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:44.128157    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4q6m" event={"ID":"4665c154-7d62-47da-b6cb-7f221aa36343","Type":"ContainerDied","Data":"b5154b3afa7c45a1bcc83d09599f49cda2f9c3d188a0fa89c152fe78645796c9"}
Sep 16 18:16:44.141333 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:44.141304    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:44.154933 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:44.154899    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7" podStartSLOduration=8.657875059 podStartE2EDuration="27.154887379s" totalImagesPullingTime="18.49701232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:44.154432037 +0000 UTC m=+27.713599873" watchObservedRunningTime="2026-09-16 18:16:44.154887379 +0000 UTC m=+27.714055209"
Sep 16 18:16:44.877568 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:44.877529    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:44.877786 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:44.877665    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:44.877853 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:44.877529    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:44.877979 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:44.877951    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:45.133171 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:45.133089    2554 generic.go:356] "Generic (PLEG): container finished" podID="4665c154-7d62-47da-b6cb-7f221aa36343" containerID="22242b326d08999f099dc51433b4a87e0138092536c0f985f0ff30f142c01429" exitCode=0
Sep 16 18:16:45.133957 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:45.133180    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4q6m" event={"ID":"4665c154-7d62-47da-b6cb-7f221aa36343","Type":"ContainerDied","Data":"22242b326d08999f099dc51433b4a87e0138092536c0f985f0ff30f142c01429"}
Sep 16 18:16:45.133957 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:45.133755    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:45.148137 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:45.148117    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:16:45.370924 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:45.370897    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vmvpp"]
Sep 16 18:16:45.371074 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:45.370993    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:45.371114 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:45.371074    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:45.373851 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:45.373819    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7w74g"]
Sep 16 18:16:45.373962 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:45.373896    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:45.374025 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:45.373977    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:45.376656 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:45.376633    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b29qk"]
Sep 16 18:16:45.376749 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:45.376706    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:45.376788 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:45.376767    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:46.137959 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:46.137876    2554 generic.go:356] "Generic (PLEG): container finished" podID="4665c154-7d62-47da-b6cb-7f221aa36343" containerID="082cc3299b476302f80d770fa622475e30abaf599f065008c0b564a886a4d280" exitCode=0
Sep 16 18:16:46.138531 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:46.137955    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4q6m" event={"ID":"4665c154-7d62-47da-b6cb-7f221aa36343","Type":"ContainerDied","Data":"082cc3299b476302f80d770fa622475e30abaf599f065008c0b564a886a4d280"}
Sep 16 18:16:46.878223 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:46.877943    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:46.878394 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:46.878035    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:46.878394 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:46.878248    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:46.878394 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:46.878349    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:46.878394 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:46.878068    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:46.878578 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:46.878435    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:48.878076 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:48.878038    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:48.878507 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:48.878038    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:48.878507 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:48.878171    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:48.878507 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:48.878284    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:48.878507 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:48.878038    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:48.878507 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:48.878396    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:50.517029 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:50.516997    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnc6b\" (UniqueName: \"kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b\") pod \"network-check-target-b29qk\" (UID: \"84e3cf97-48f5-415e-a5f5-a4cf55bda177\") " pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:50.517394 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:50.517130    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:16:50.517394 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:50.517145    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:16:50.517394 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:50.517154    2554 projected.go:196] Error preparing data for projected volume kube-api-access-mnc6b for pod openshift-network-diagnostics/network-check-target-b29qk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:50.517394 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:50.517199    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b podName:84e3cf97-48f5-415e-a5f5-a4cf55bda177 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:22.51718578 +0000 UTC m=+66.076353605 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnc6b" (UniqueName: "kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b") pod "network-check-target-b29qk" (UID: "84e3cf97-48f5-415e-a5f5-a4cf55bda177") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:50.617478 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:50.617422    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs\") pod \"network-metrics-daemon-vmvpp\" (UID: \"e4c7d4e4-64cb-4cdd-90ae-db41d1889099\") " pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:50.617586 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:50.617483    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret\") pod \"global-pull-secret-syncer-7w74g\" (UID: \"50b4667a-4d35-44cb-865b-747a72c7c3e6\") " pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:50.617586 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:50.617570    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:50.617683 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:50.617602    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:50.617683 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:50.617636    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs podName:e4c7d4e4-64cb-4cdd-90ae-db41d1889099 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:22.617620102 +0000 UTC m=+66.176787916 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs") pod "network-metrics-daemon-vmvpp" (UID: "e4c7d4e4-64cb-4cdd-90ae-db41d1889099") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:50.617683 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:50.617651    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret podName:50b4667a-4d35-44cb-865b-747a72c7c3e6 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:22.617645228 +0000 UTC m=+66.176813040 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret") pod "global-pull-secret-syncer-7w74g" (UID: "50b4667a-4d35-44cb-865b-747a72c7c3e6") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:50.878289 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:50.878184    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:50.878289 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:50.878184    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:50.878511 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:50.878330    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:50.878511 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:50.878328    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmvpp" podUID="e4c7d4e4-64cb-4cdd-90ae-db41d1889099"
Sep 16 18:16:50.878511 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:50.878413    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w74g" podUID="50b4667a-4d35-44cb-865b-747a72c7c3e6"
Sep 16 18:16:50.878625 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:50.878508    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b29qk" podUID="84e3cf97-48f5-415e-a5f5-a4cf55bda177"
Sep 16 18:16:51.153538 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.153471    2554 generic.go:356] "Generic (PLEG): container finished" podID="4665c154-7d62-47da-b6cb-7f221aa36343" containerID="4507aa80cffd1ab677f6528abacbb3fad9665bc98aad803693d1e07788d10707" exitCode=0
Sep 16 18:16:51.153538 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.153520    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4q6m" event={"ID":"4665c154-7d62-47da-b6cb-7f221aa36343","Type":"ContainerDied","Data":"4507aa80cffd1ab677f6528abacbb3fad9665bc98aad803693d1e07788d10707"}
Sep 16 18:16:51.238821 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.238793    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeReady"
Sep 16 18:16:51.238928 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.238913    2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:16:51.276090 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.276063    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cfb89dbc-l9d7f"]
Sep 16 18:16:51.289680 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.289659    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pbwcp"]
Sep 16 18:16:51.289784 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.289764    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.291964 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.291941    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:16:51.292066 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.292034    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:16:51.292128 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.292094    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mlk5x\""
Sep 16 18:16:51.292128 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.292115    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:16:51.302884 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.302824    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:16:51.308409 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.308388    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cfb89dbc-l9d7f"]
Sep 16 18:16:51.308507 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.308419    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zs8q4"]
Sep 16 18:16:51.308571 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.308513    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:16:51.310771 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.310751    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:16:51.310866 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.310776    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:16:51.310866 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.310808    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f8sdc\""
Sep 16 18:16:51.328168 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.328148    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pbwcp"]
Sep 16 18:16:51.328263 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.328178    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zs8q4"]
Sep 16 18:16:51.328307 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.328265    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zs8q4"
Sep 16 18:16:51.331148 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.331121    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:16:51.331148 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.331142    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:16:51.331315 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.331205    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:16:51.331315 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.331240    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zhggq\""
Sep 16 18:16:51.423684 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.423626    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9812ded9-6393-45db-9c21-338afa68c6b9-registry-certificates\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.423684 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.423658    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9812ded9-6393-45db-9c21-338afa68c6b9-trusted-ca\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.423684 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.423683    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90a6aab5-3c38-4775-ba9c-3dc99fc92207-config-volume\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:16:51.423908 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.423705    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjv2\" (UniqueName: \"kubernetes.io/projected/90a6aab5-3c38-4775-ba9c-3dc99fc92207-kube-api-access-pqjv2\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:16:51.423908 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.423771    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9812ded9-6393-45db-9c21-338afa68c6b9-installation-pull-secrets\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.423908 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.423807    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a6aab5-3c38-4775-ba9c-3dc99fc92207-tmp-dir\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:16:51.423908 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.423839    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmntm\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-kube-api-access-rmntm\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.423908 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.423867    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-bound-sa-token\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.423908 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.423901    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:16:51.424162 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.423932    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9812ded9-6393-45db-9c21-338afa68c6b9-image-registry-private-configuration\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.424162 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.423953    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxgzq\" (UniqueName: \"kubernetes.io/projected/9569d7bb-5ebe-4d49-8a92-faed5784b8c7-kube-api-access-cxgzq\") pod \"ingress-canary-zs8q4\" (UID: \"9569d7bb-5ebe-4d49-8a92-faed5784b8c7\") " pod="openshift-ingress-canary/ingress-canary-zs8q4"
Sep 16 18:16:51.424162 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.423982    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9812ded9-6393-45db-9c21-338afa68c6b9-ca-trust-extracted\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.424162 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.423998    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.424162 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.424026    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9569d7bb-5ebe-4d49-8a92-faed5784b8c7-cert\") pod \"ingress-canary-zs8q4\" (UID: \"9569d7bb-5ebe-4d49-8a92-faed5784b8c7\") " pod="openshift-ingress-canary/ingress-canary-zs8q4"
Sep 16 18:16:51.524425 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.524391    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9812ded9-6393-45db-9c21-338afa68c6b9-registry-certificates\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.524425 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.524424    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9812ded9-6393-45db-9c21-338afa68c6b9-trusted-ca\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.524881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.524465    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90a6aab5-3c38-4775-ba9c-3dc99fc92207-config-volume\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:16:51.524881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.524489    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjv2\" (UniqueName: \"kubernetes.io/projected/90a6aab5-3c38-4775-ba9c-3dc99fc92207-kube-api-access-pqjv2\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:16:51.524881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.524516    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9812ded9-6393-45db-9c21-338afa68c6b9-installation-pull-secrets\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.524881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.524546    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a6aab5-3c38-4775-ba9c-3dc99fc92207-tmp-dir\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:16:51.524881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.524575    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmntm\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-kube-api-access-rmntm\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.524881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.524601    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-bound-sa-token\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.524881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.524634    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:16:51.524881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.524664    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9812ded9-6393-45db-9c21-338afa68c6b9-image-registry-private-configuration\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.524881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.524689    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxgzq\" (UniqueName: \"kubernetes.io/projected/9569d7bb-5ebe-4d49-8a92-faed5784b8c7-kube-api-access-cxgzq\") pod \"ingress-canary-zs8q4\" (UID: \"9569d7bb-5ebe-4d49-8a92-faed5784b8c7\") " pod="openshift-ingress-canary/ingress-canary-zs8q4"
Sep 16 18:16:51.524881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.524728    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9812ded9-6393-45db-9c21-338afa68c6b9-ca-trust-extracted\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.524881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.524750    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.524881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.524768    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9569d7bb-5ebe-4d49-8a92-faed5784b8c7-cert\") pod \"ingress-canary-zs8q4\" (UID: \"9569d7bb-5ebe-4d49-8a92-faed5784b8c7\") " pod="openshift-ingress-canary/ingress-canary-zs8q4"
Sep 16 18:16:51.525515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.525140    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a6aab5-3c38-4775-ba9c-3dc99fc92207-tmp-dir\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:16:51.525515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.525336    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90a6aab5-3c38-4775-ba9c-3dc99fc92207-config-volume\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:16:51.525515 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:51.525435    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:16:51.525515 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:51.525512    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls podName:90a6aab5-3c38-4775-ba9c-3dc99fc92207 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:52.025494324 +0000 UTC m=+35.584662138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls") pod "dns-default-pbwcp" (UID: "90a6aab5-3c38-4775-ba9c-3dc99fc92207") : secret "dns-default-metrics-tls" not found
Sep 16 18:16:51.525727 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.525550    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9812ded9-6393-45db-9c21-338afa68c6b9-ca-trust-extracted\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.525727 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:51.525639    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:16:51.525727 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:51.525652    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfb89dbc-l9d7f: secret "image-registry-tls" not found
Sep 16 18:16:51.525727 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:51.525693    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls podName:9812ded9-6393-45db-9c21-338afa68c6b9 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:52.02567846 +0000 UTC m=+35.584846275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls") pod "image-registry-6cfb89dbc-l9d7f" (UID: "9812ded9-6393-45db-9c21-338afa68c6b9") : secret "image-registry-tls" not found
Sep 16 18:16:51.525900 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.525806    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9812ded9-6393-45db-9c21-338afa68c6b9-trusted-ca\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.526015 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.525994    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9812ded9-6393-45db-9c21-338afa68c6b9-registry-certificates\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.528717 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.528694    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9812ded9-6393-45db-9c21-338afa68c6b9-installation-pull-secrets\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.528717 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.528714    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9569d7bb-5ebe-4d49-8a92-faed5784b8c7-cert\") pod \"ingress-canary-zs8q4\" (UID: \"9569d7bb-5ebe-4d49-8a92-faed5784b8c7\") " pod="openshift-ingress-canary/ingress-canary-zs8q4"
Sep 16 18:16:51.528836 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.528701    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9812ded9-6393-45db-9c21-338afa68c6b9-image-registry-private-configuration\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.536480 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.536456    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxgzq\" (UniqueName: \"kubernetes.io/projected/9569d7bb-5ebe-4d49-8a92-faed5784b8c7-kube-api-access-cxgzq\") pod \"ingress-canary-zs8q4\" (UID: \"9569d7bb-5ebe-4d49-8a92-faed5784b8c7\") " pod="openshift-ingress-canary/ingress-canary-zs8q4"
Sep 16 18:16:51.538299 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.538281    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjv2\" (UniqueName: \"kubernetes.io/projected/90a6aab5-3c38-4775-ba9c-3dc99fc92207-kube-api-access-pqjv2\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:16:51.540993 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.540973    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-bound-sa-token\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.541611 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.541590    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmntm\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-kube-api-access-rmntm\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:51.636018 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.635991    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zs8q4"
Sep 16 18:16:51.961600 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:51.961573    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zs8q4"]
Sep 16 18:16:51.964633 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:16:51.964601    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9569d7bb_5ebe_4d49_8a92_faed5784b8c7.slice/crio-d8b80e7baa0b256325621d10fbfa66d4be6fba5d4de53237ed5947b51ed28cb6 WatchSource:0}: Error finding container d8b80e7baa0b256325621d10fbfa66d4be6fba5d4de53237ed5947b51ed28cb6: Status 404 returned error can't find the container with id d8b80e7baa0b256325621d10fbfa66d4be6fba5d4de53237ed5947b51ed28cb6
Sep 16 18:16:52.028059 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:52.028005    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:16:52.028059 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:52.028047    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:52.028176 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:52.028143    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:16:52.028207 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:52.028192    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:16:52.028207 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:52.028203    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfb89dbc-l9d7f: secret "image-registry-tls" not found
Sep 16 18:16:52.028325 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:52.028196    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls podName:90a6aab5-3c38-4775-ba9c-3dc99fc92207 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:53.028182277 +0000 UTC m=+36.587350103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls") pod "dns-default-pbwcp" (UID: "90a6aab5-3c38-4775-ba9c-3dc99fc92207") : secret "dns-default-metrics-tls" not found
Sep 16 18:16:52.028325 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:52.028268    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls podName:9812ded9-6393-45db-9c21-338afa68c6b9 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:53.028251869 +0000 UTC m=+36.587419685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls") pod "image-registry-6cfb89dbc-l9d7f" (UID: "9812ded9-6393-45db-9c21-338afa68c6b9") : secret "image-registry-tls" not found
Sep 16 18:16:52.158769 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:52.158740    2554 generic.go:356] "Generic (PLEG): container finished" podID="4665c154-7d62-47da-b6cb-7f221aa36343" containerID="5a758dfe87ed7271520f2c5decd8c7afe57c5a6110200b24fc5e9f4c70e20bc6" exitCode=0
Sep 16 18:16:52.158917 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:52.158813    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4q6m" event={"ID":"4665c154-7d62-47da-b6cb-7f221aa36343","Type":"ContainerDied","Data":"5a758dfe87ed7271520f2c5decd8c7afe57c5a6110200b24fc5e9f4c70e20bc6"}
Sep 16 18:16:52.160010 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:52.159991    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zs8q4" event={"ID":"9569d7bb-5ebe-4d49-8a92-faed5784b8c7","Type":"ContainerStarted","Data":"d8b80e7baa0b256325621d10fbfa66d4be6fba5d4de53237ed5947b51ed28cb6"}
Sep 16 18:16:52.877512 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:52.877474    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:16:52.878496 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:52.877474    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:16:52.878496 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:52.877792    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:16:52.880479 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:52.880454    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:16:52.881099 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:52.880720    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:16:52.881099 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:52.880742    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5rgdl\""
Sep 16 18:16:52.881099 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:52.880829    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:16:52.881099 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:52.880937    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bfgtt\""
Sep 16 18:16:52.881099 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:52.880988    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:16:53.038750 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:53.038715    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:16:53.038907 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:53.038886    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:16:53.038982 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:53.038940    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:53.038982 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:53.038961    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls podName:90a6aab5-3c38-4775-ba9c-3dc99fc92207 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:55.038940788 +0000 UTC m=+38.598108615 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls") pod "dns-default-pbwcp" (UID: "90a6aab5-3c38-4775-ba9c-3dc99fc92207") : secret "dns-default-metrics-tls" not found
Sep 16 18:16:53.039110 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:53.039015    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:16:53.039110 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:53.039025    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfb89dbc-l9d7f: secret "image-registry-tls" not found
Sep 16 18:16:53.039110 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:53.039072    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls podName:9812ded9-6393-45db-9c21-338afa68c6b9 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:55.039051916 +0000 UTC m=+38.598219728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls") pod "image-registry-6cfb89dbc-l9d7f" (UID: "9812ded9-6393-45db-9c21-338afa68c6b9") : secret "image-registry-tls" not found
Sep 16 18:16:53.167219 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:53.167135    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4q6m" event={"ID":"4665c154-7d62-47da-b6cb-7f221aa36343","Type":"ContainerStarted","Data":"4f708255cb590be932683eb3a7d72ee5bd547093b25e9b2e8bf3f80d91054ae8"}
Sep 16 18:16:53.188040 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:53.187990    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v4q6m" podStartSLOduration=7.578321539 podStartE2EDuration="36.187974731s" totalImagesPullingTime="28.609653192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:53.187541744 +0000 UTC m=+36.746709592" watchObservedRunningTime="2026-09-16 18:16:53.187974731 +0000 UTC m=+36.747142564"
Sep 16 18:16:55.056226 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:55.056038    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:16:55.056669 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:55.056275    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:55.056669 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:55.056187    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:16:55.056669 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:55.056365    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls podName:90a6aab5-3c38-4775-ba9c-3dc99fc92207 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:59.056341941 +0000 UTC m=+42.615509776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls") pod "dns-default-pbwcp" (UID: "90a6aab5-3c38-4775-ba9c-3dc99fc92207") : secret "dns-default-metrics-tls" not found
Sep 16 18:16:55.056669 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:55.056395    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:16:55.056669 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:55.056405    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfb89dbc-l9d7f: secret "image-registry-tls" not found
Sep 16 18:16:55.056669 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:55.056441    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls podName:9812ded9-6393-45db-9c21-338afa68c6b9 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:59.056430884 +0000 UTC m=+42.615598699 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls") pod "image-registry-6cfb89dbc-l9d7f" (UID: "9812ded9-6393-45db-9c21-338afa68c6b9") : secret "image-registry-tls" not found
Sep 16 18:16:55.172791 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:55.172715    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zs8q4" event={"ID":"9569d7bb-5ebe-4d49-8a92-faed5784b8c7","Type":"ContainerStarted","Data":"c00cf07985fc008b90180b312fd27ab9af09eda8b18ba90fad8f6fa8ac2562ca"}
Sep 16 18:16:55.192847 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:55.192804    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zs8q4" podStartSLOduration=1.7765677260000001 podStartE2EDuration="4.192790592s" totalImagesPullingTime="2.416222866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:55.191303382 +0000 UTC m=+38.750471215" watchObservedRunningTime="2026-09-16 18:16:55.192790592 +0000 UTC m=+38.751958426"
Sep 16 18:16:59.087338 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:59.087303    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:16:59.087699 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:16:59.087353    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:16:59.087699 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:59.087461    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:16:59.087699 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:59.087526    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:16:59.087699 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:59.087543    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfb89dbc-l9d7f: secret "image-registry-tls" not found
Sep 16 18:16:59.087699 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:59.087534    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls podName:90a6aab5-3c38-4775-ba9c-3dc99fc92207 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:07.087519016 +0000 UTC m=+50.646686827 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls") pod "dns-default-pbwcp" (UID: "90a6aab5-3c38-4775-ba9c-3dc99fc92207") : secret "dns-default-metrics-tls" not found
Sep 16 18:16:59.087699 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:16:59.087604    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls podName:9812ded9-6393-45db-9c21-338afa68c6b9 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:07.08758579 +0000 UTC m=+50.646753611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls") pod "image-registry-6cfb89dbc-l9d7f" (UID: "9812ded9-6393-45db-9c21-338afa68c6b9") : secret "image-registry-tls" not found
Sep 16 18:17:07.138757 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:07.138720    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:17:07.139171 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:07.138771    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:17:07.139171 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:07.138871    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:17:07.139171 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:07.138945    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls podName:90a6aab5-3c38-4775-ba9c-3dc99fc92207 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:23.13892688 +0000 UTC m=+66.698094691 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls") pod "dns-default-pbwcp" (UID: "90a6aab5-3c38-4775-ba9c-3dc99fc92207") : secret "dns-default-metrics-tls" not found
Sep 16 18:17:07.139171 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:07.138877    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:17:07.139171 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:07.138969    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfb89dbc-l9d7f: secret "image-registry-tls" not found
Sep 16 18:17:07.139171 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:07.139019    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls podName:9812ded9-6393-45db-9c21-338afa68c6b9 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:23.139008129 +0000 UTC m=+66.698175941 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls") pod "image-registry-6cfb89dbc-l9d7f" (UID: "9812ded9-6393-45db-9c21-338afa68c6b9") : secret "image-registry-tls" not found
Sep 16 18:17:17.150673 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.150647    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvzv7"
Sep 16 18:17:17.224465 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.224426    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-bz98l"]
Sep 16 18:17:17.226956 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.226940    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:17.230310 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.230290    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 18:17:17.231179 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.231165    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:17:17.231635 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.231618    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 18:17:17.231942 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.231916    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5zll9\""
Sep 16 18:17:17.232304 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.232286    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 18:17:17.235928 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.235912    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 18:17:17.246134 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.246114    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-bz98l"]
Sep 16 18:17:17.305333 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.305297    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aab214d7-cd3b-4018-b947-4e6c626eea04-config\") pod \"console-operator-696884fb9c-bz98l\" (UID: \"aab214d7-cd3b-4018-b947-4e6c626eea04\") " pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:17.305494 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.305338    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aab214d7-cd3b-4018-b947-4e6c626eea04-serving-cert\") pod \"console-operator-696884fb9c-bz98l\" (UID: \"aab214d7-cd3b-4018-b947-4e6c626eea04\") " pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:17.305494 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.305366    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aab214d7-cd3b-4018-b947-4e6c626eea04-trusted-ca\") pod \"console-operator-696884fb9c-bz98l\" (UID: \"aab214d7-cd3b-4018-b947-4e6c626eea04\") " pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:17.305494 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.305432    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbtp5\" (UniqueName: \"kubernetes.io/projected/aab214d7-cd3b-4018-b947-4e6c626eea04-kube-api-access-rbtp5\") pod \"console-operator-696884fb9c-bz98l\" (UID: \"aab214d7-cd3b-4018-b947-4e6c626eea04\") " pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:17.315863 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.315838    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-gwsht"]
Sep 16 18:17:17.317435 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.317421    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-gwsht"
Sep 16 18:17:17.320110 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.320094    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-b9rw7\""
Sep 16 18:17:17.332720 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.332696    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-gwsht"]
Sep 16 18:17:17.405740 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.405660    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aab214d7-cd3b-4018-b947-4e6c626eea04-trusted-ca\") pod \"console-operator-696884fb9c-bz98l\" (UID: \"aab214d7-cd3b-4018-b947-4e6c626eea04\") " pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:17.405740 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.405708    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbtp5\" (UniqueName: \"kubernetes.io/projected/aab214d7-cd3b-4018-b947-4e6c626eea04-kube-api-access-rbtp5\") pod \"console-operator-696884fb9c-bz98l\" (UID: \"aab214d7-cd3b-4018-b947-4e6c626eea04\") " pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:17.405932 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.405775    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8stxz\" (UniqueName: \"kubernetes.io/projected/da2af82b-7352-4739-92c7-b38bfe98c266-kube-api-access-8stxz\") pod \"network-check-source-6f8b768455-gwsht\" (UID: \"da2af82b-7352-4739-92c7-b38bfe98c266\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-gwsht"
Sep 16 18:17:17.405932 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.405793    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aab214d7-cd3b-4018-b947-4e6c626eea04-config\") pod \"console-operator-696884fb9c-bz98l\" (UID: \"aab214d7-cd3b-4018-b947-4e6c626eea04\") " pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:17.405932 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.405813    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aab214d7-cd3b-4018-b947-4e6c626eea04-serving-cert\") pod \"console-operator-696884fb9c-bz98l\" (UID: \"aab214d7-cd3b-4018-b947-4e6c626eea04\") " pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:17.406388 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.406365    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aab214d7-cd3b-4018-b947-4e6c626eea04-config\") pod \"console-operator-696884fb9c-bz98l\" (UID: \"aab214d7-cd3b-4018-b947-4e6c626eea04\") " pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:17.406677 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.406660    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aab214d7-cd3b-4018-b947-4e6c626eea04-trusted-ca\") pod \"console-operator-696884fb9c-bz98l\" (UID: \"aab214d7-cd3b-4018-b947-4e6c626eea04\") " pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:17.408952 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.408931    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aab214d7-cd3b-4018-b947-4e6c626eea04-serving-cert\") pod \"console-operator-696884fb9c-bz98l\" (UID: \"aab214d7-cd3b-4018-b947-4e6c626eea04\") " pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:17.419370 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.419342    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbtp5\" (UniqueName: \"kubernetes.io/projected/aab214d7-cd3b-4018-b947-4e6c626eea04-kube-api-access-rbtp5\") pod \"console-operator-696884fb9c-bz98l\" (UID: \"aab214d7-cd3b-4018-b947-4e6c626eea04\") " pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:17.506896 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.506846    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8stxz\" (UniqueName: \"kubernetes.io/projected/da2af82b-7352-4739-92c7-b38bfe98c266-kube-api-access-8stxz\") pod \"network-check-source-6f8b768455-gwsht\" (UID: \"da2af82b-7352-4739-92c7-b38bfe98c266\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-gwsht"
Sep 16 18:17:17.515116 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.515096    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:17:17.526188 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.526170    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:17:17.535476 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.535446    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:17.536669 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.536629    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8stxz\" (UniqueName: \"kubernetes.io/projected/da2af82b-7352-4739-92c7-b38bfe98c266-kube-api-access-8stxz\") pod \"network-check-source-6f8b768455-gwsht\" (UID: \"da2af82b-7352-4739-92c7-b38bfe98c266\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-gwsht"
Sep 16 18:17:17.625245 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.625198    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-gwsht"
Sep 16 18:17:17.804124 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.804090    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-bz98l"]
Sep 16 18:17:17.807111 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:17:17.807086    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab214d7_cd3b_4018_b947_4e6c626eea04.slice/crio-93bd898bce09e34b1a6e3c75aa5ec9123a17b5b54a95ae36fadda6f8d56d9d45 WatchSource:0}: Error finding container 93bd898bce09e34b1a6e3c75aa5ec9123a17b5b54a95ae36fadda6f8d56d9d45: Status 404 returned error can't find the container with id 93bd898bce09e34b1a6e3c75aa5ec9123a17b5b54a95ae36fadda6f8d56d9d45
Sep 16 18:17:17.893862 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:17.893831    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-gwsht"]
Sep 16 18:17:17.896938 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:17:17.896909    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2af82b_7352_4739_92c7_b38bfe98c266.slice/crio-3a983e1129a5bfbd6482ec707c67d990ea7ea3aba137faf49034cf66df50f487 WatchSource:0}: Error finding container 3a983e1129a5bfbd6482ec707c67d990ea7ea3aba137faf49034cf66df50f487: Status 404 returned error can't find the container with id 3a983e1129a5bfbd6482ec707c67d990ea7ea3aba137faf49034cf66df50f487
Sep 16 18:17:18.224714 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:18.224675    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-gwsht" event={"ID":"da2af82b-7352-4739-92c7-b38bfe98c266","Type":"ContainerStarted","Data":"3a983e1129a5bfbd6482ec707c67d990ea7ea3aba137faf49034cf66df50f487"}
Sep 16 18:17:18.225843 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:18.225822    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bz98l" event={"ID":"aab214d7-cd3b-4018-b947-4e6c626eea04","Type":"ContainerStarted","Data":"93bd898bce09e34b1a6e3c75aa5ec9123a17b5b54a95ae36fadda6f8d56d9d45"}
Sep 16 18:17:22.149448 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.149416    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5z599"]
Sep 16 18:17:22.168660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.168634    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-cvns7"]
Sep 16 18:17:22.168833 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.168776    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:22.171202 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.171180    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:17:22.171436 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.171218    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:17:22.171436 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.171273    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:17:22.171608 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.171337    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:17:22.172076 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.172059    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9v5nj\""
Sep 16 18:17:22.195422 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.195399    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5z599"]
Sep 16 18:17:22.195537 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.195445    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-cvns7"]
Sep 16 18:17:22.195590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.195538    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cvns7"
Sep 16 18:17:22.198132 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.198105    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:17:22.198132 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.198125    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:17:22.198306 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.198106    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4lxcr\""
Sep 16 18:17:22.236703 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.236667    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-gwsht" event={"ID":"da2af82b-7352-4739-92c7-b38bfe98c266","Type":"ContainerStarted","Data":"ae8ac4ae4b3cd7f1030c25ddca465ad4d84e855d20b2f86d431b8f21992d5fcf"}
Sep 16 18:17:22.238332 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.238306    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/0.log"
Sep 16 18:17:22.238475 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.238359    2554 generic.go:356] "Generic (PLEG): container finished" podID="aab214d7-cd3b-4018-b947-4e6c626eea04" containerID="1f42e5a31dad2002c86f52575ce769d531dd9e1e7096d3fd62a200fe0150b613" exitCode=255
Sep 16 18:17:22.238543 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.238507    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bz98l" event={"ID":"aab214d7-cd3b-4018-b947-4e6c626eea04","Type":"ContainerDied","Data":"1f42e5a31dad2002c86f52575ce769d531dd9e1e7096d3fd62a200fe0150b613"}
Sep 16 18:17:22.238646 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.238634    2554 scope.go:117] "RemoveContainer" containerID="1f42e5a31dad2002c86f52575ce769d531dd9e1e7096d3fd62a200fe0150b613"
Sep 16 18:17:22.245246 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.245210    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-data-volume\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:22.245352 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.245259    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:22.245352 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.245290    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vzvw\" (UniqueName: \"kubernetes.io/projected/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-api-access-4vzvw\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:22.245352 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.245335    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:22.245464 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.245383    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-crio-socket\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:22.245464 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.245405    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn9nx\" (UniqueName: \"kubernetes.io/projected/9f55b4e3-4cf9-4ddb-895c-5cdde11c2373-kube-api-access-fn9nx\") pod \"migrator-8b4d697c5-cvns7\" (UID: \"9f55b4e3-4cf9-4ddb-895c-5cdde11c2373\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cvns7"
Sep 16 18:17:22.256792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.256744    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-gwsht" podStartSLOduration=1.9282771410000001 podStartE2EDuration="5.256729635s" totalImagesPullingTime="3.328452494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:22.255768128 +0000 UTC m=+65.814935961" watchObservedRunningTime="2026-09-16 18:17:22.256729635 +0000 UTC m=+65.815897469"
Sep 16 18:17:22.346403 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.346351    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:22.346595 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.346430    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-crio-socket\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:22.346595 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.346455    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn9nx\" (UniqueName: \"kubernetes.io/projected/9f55b4e3-4cf9-4ddb-895c-5cdde11c2373-kube-api-access-fn9nx\") pod \"migrator-8b4d697c5-cvns7\" (UID: \"9f55b4e3-4cf9-4ddb-895c-5cdde11c2373\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cvns7"
Sep 16 18:17:22.346595 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:22.346514    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:17:22.346595 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.346545    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-crio-socket\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:22.346595 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:22.346581    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm podName:f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:22.846566059 +0000 UTC m=+66.405733870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5z599" (UID: "f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1") : configmap "kube-rbac-proxy" not found
Sep 16 18:17:22.346920 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.346600    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-data-volume\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:22.346920 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.346621    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:22.346920 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.346651    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vzvw\" (UniqueName: \"kubernetes.io/projected/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-api-access-4vzvw\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:22.347069 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.347058    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-data-volume\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:22.347121 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:22.347081    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:17:22.347168 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:22.347134    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls podName:f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:22.847118093 +0000 UTC m=+66.406285904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5z599" (UID: "f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:17:22.356670 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.356639    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn9nx\" (UniqueName: \"kubernetes.io/projected/9f55b4e3-4cf9-4ddb-895c-5cdde11c2373-kube-api-access-fn9nx\") pod \"migrator-8b4d697c5-cvns7\" (UID: \"9f55b4e3-4cf9-4ddb-895c-5cdde11c2373\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cvns7"
Sep 16 18:17:22.356787 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.356703    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vzvw\" (UniqueName: \"kubernetes.io/projected/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-api-access-4vzvw\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:22.504218 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.504181    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cvns7"
Sep 16 18:17:22.548581 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.548538    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnc6b\" (UniqueName: \"kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b\") pod \"network-check-target-b29qk\" (UID: \"84e3cf97-48f5-415e-a5f5-a4cf55bda177\") " pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:17:22.550980 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.550954    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnc6b\" (UniqueName: \"kubernetes.io/projected/84e3cf97-48f5-415e-a5f5-a4cf55bda177-kube-api-access-mnc6b\") pod \"network-check-target-b29qk\" (UID: \"84e3cf97-48f5-415e-a5f5-a4cf55bda177\") " pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:17:22.596383 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.596357    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bfgtt\""
Sep 16 18:17:22.603778 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.603753    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:17:22.649742 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.649293    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs\") pod \"network-metrics-daemon-vmvpp\" (UID: \"e4c7d4e4-64cb-4cdd-90ae-db41d1889099\") " pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:17:22.649742 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.649423    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret\") pod \"global-pull-secret-syncer-7w74g\" (UID: \"50b4667a-4d35-44cb-865b-747a72c7c3e6\") " pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:17:22.652378 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.652352    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:17:22.652771 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.652354    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:17:22.660484 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:22.659962    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:17:22.660484 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:22.660057    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs podName:e4c7d4e4-64cb-4cdd-90ae-db41d1889099 nodeName:}" failed. No retries permitted until 2026-09-16 18:18:26.660032211 +0000 UTC m=+130.219200035 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs") pod "network-metrics-daemon-vmvpp" (UID: "e4c7d4e4-64cb-4cdd-90ae-db41d1889099") : secret "metrics-daemon-secret" not found
Sep 16 18:17:22.662479 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.662456    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/50b4667a-4d35-44cb-865b-747a72c7c3e6-original-pull-secret\") pod \"global-pull-secret-syncer-7w74g\" (UID: \"50b4667a-4d35-44cb-865b-747a72c7c3e6\") " pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:17:22.777376 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.777293    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-cvns7"]
Sep 16 18:17:22.781166 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:17:22.781138    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f55b4e3_4cf9_4ddb_895c_5cdde11c2373.slice/crio-62579bc1fac4595a82729a77548216b9aaeac5804abd1cd261317cdf44146cf6 WatchSource:0}: Error finding container 62579bc1fac4595a82729a77548216b9aaeac5804abd1cd261317cdf44146cf6: Status 404 returned error can't find the container with id 62579bc1fac4595a82729a77548216b9aaeac5804abd1cd261317cdf44146cf6
Sep 16 18:17:22.851469 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.851430    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:22.851659 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.851544    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:22.851659 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:22.851557    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:17:22.851659 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:22.851616    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls podName:f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:23.851595256 +0000 UTC m=+67.410763069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5z599" (UID: "f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:17:22.851659 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:22.851638    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:17:22.851887 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:22.851678    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm podName:f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:23.851667421 +0000 UTC m=+67.410835236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5z599" (UID: "f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1") : configmap "kube-rbac-proxy" not found
Sep 16 18:17:22.883252 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.883209    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b29qk"]
Sep 16 18:17:22.887498 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:17:22.887472    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e3cf97_48f5_415e_a5f5_a4cf55bda177.slice/crio-999aa043961a8ab067462d5713a356e4fe9eefb2c00cae42fb02deef22a876c5 WatchSource:0}: Error finding container 999aa043961a8ab067462d5713a356e4fe9eefb2c00cae42fb02deef22a876c5: Status 404 returned error can't find the container with id 999aa043961a8ab067462d5713a356e4fe9eefb2c00cae42fb02deef22a876c5
Sep 16 18:17:22.901412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:22.901385    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w74g"
Sep 16 18:17:23.154127 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.154053    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:17:23.154127 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.154109    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:17:23.154557 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:23.154192    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:17:23.154557 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:23.154253    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:17:23.154557 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:23.154266    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls podName:90a6aab5-3c38-4775-ba9c-3dc99fc92207 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:55.154250222 +0000 UTC m=+98.713418033 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls") pod "dns-default-pbwcp" (UID: "90a6aab5-3c38-4775-ba9c-3dc99fc92207") : secret "dns-default-metrics-tls" not found
Sep 16 18:17:23.154557 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:23.154270    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfb89dbc-l9d7f: secret "image-registry-tls" not found
Sep 16 18:17:23.154557 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:23.154320    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls podName:9812ded9-6393-45db-9c21-338afa68c6b9 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:55.154304756 +0000 UTC m=+98.713472586 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls") pod "image-registry-6cfb89dbc-l9d7f" (UID: "9812ded9-6393-45db-9c21-338afa68c6b9") : secret "image-registry-tls" not found
Sep 16 18:17:23.172263 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.172227    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7w74g"]
Sep 16 18:17:23.174546 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:17:23.174518    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b4667a_4d35_44cb_865b_747a72c7c3e6.slice/crio-73c0e57ee34d31ee3208184f0798654e45f75fb6e0a87216de867f6edac32dc2 WatchSource:0}: Error finding container 73c0e57ee34d31ee3208184f0798654e45f75fb6e0a87216de867f6edac32dc2: Status 404 returned error can't find the container with id 73c0e57ee34d31ee3208184f0798654e45f75fb6e0a87216de867f6edac32dc2
Sep 16 18:17:23.242793 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.242768    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/1.log"
Sep 16 18:17:23.243165 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.243147    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/0.log"
Sep 16 18:17:23.243213 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.243185    2554 generic.go:356] "Generic (PLEG): container finished" podID="aab214d7-cd3b-4018-b947-4e6c626eea04" containerID="5bb50fab82e116d0d9c5536a5d24abf6bde7bb0ee0dd450f9595fabb6f910a3e" exitCode=255
Sep 16 18:17:23.243291 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.243262    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bz98l" event={"ID":"aab214d7-cd3b-4018-b947-4e6c626eea04","Type":"ContainerDied","Data":"5bb50fab82e116d0d9c5536a5d24abf6bde7bb0ee0dd450f9595fabb6f910a3e"}
Sep 16 18:17:23.243395 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.243325    2554 scope.go:117] "RemoveContainer" containerID="1f42e5a31dad2002c86f52575ce769d531dd9e1e7096d3fd62a200fe0150b613"
Sep 16 18:17:23.243566 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.243550    2554 scope.go:117] "RemoveContainer" containerID="5bb50fab82e116d0d9c5536a5d24abf6bde7bb0ee0dd450f9595fabb6f910a3e"
Sep 16 18:17:23.243734 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:23.243716    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-bz98l_openshift-console-operator(aab214d7-cd3b-4018-b947-4e6c626eea04)\"" pod="openshift-console-operator/console-operator-696884fb9c-bz98l" podUID="aab214d7-cd3b-4018-b947-4e6c626eea04"
Sep 16 18:17:23.244824 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.244798    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b29qk" event={"ID":"84e3cf97-48f5-415e-a5f5-a4cf55bda177","Type":"ContainerStarted","Data":"8d3759468bee89ce2de99e41a5e9161f6fe609eab0c2650a8b7fb9aa3827bd9f"}
Sep 16 18:17:23.244929 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.244841    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b29qk" event={"ID":"84e3cf97-48f5-415e-a5f5-a4cf55bda177","Type":"ContainerStarted","Data":"999aa043961a8ab067462d5713a356e4fe9eefb2c00cae42fb02deef22a876c5"}
Sep 16 18:17:23.244929 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.244902    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:17:23.245944 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.245925    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7w74g" event={"ID":"50b4667a-4d35-44cb-865b-747a72c7c3e6","Type":"ContainerStarted","Data":"73c0e57ee34d31ee3208184f0798654e45f75fb6e0a87216de867f6edac32dc2"}
Sep 16 18:17:23.247001 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.246978    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cvns7" event={"ID":"9f55b4e3-4cf9-4ddb-895c-5cdde11c2373","Type":"ContainerStarted","Data":"62579bc1fac4595a82729a77548216b9aaeac5804abd1cd261317cdf44146cf6"}
Sep 16 18:17:23.282892 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.282848    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b29qk" podStartSLOduration=66.282832412 podStartE2EDuration="1m6.282832412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:23.28274409 +0000 UTC m=+66.841911945" watchObservedRunningTime="2026-09-16 18:17:23.282832412 +0000 UTC m=+66.842000244"
Sep 16 18:17:23.856626 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.856563    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dqj4w_d52c2148-fdca-4bb1-9ef7-b2492c8b7b57/dns-node-resolver/0.log"
Sep 16 18:17:23.860206 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.860175    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:23.860353 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:23.860300    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:23.860353 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:23.860322    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:17:23.860468 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:23.860386    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm podName:f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:25.860367932 +0000 UTC m=+69.419535759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5z599" (UID: "f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1") : configmap "kube-rbac-proxy" not found
Sep 16 18:17:23.860468 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:23.860433    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:17:23.860577 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:23.860488    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls podName:f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:25.860472924 +0000 UTC m=+69.419640738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5z599" (UID: "f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:17:24.252052 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:24.252021    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/1.log"
Sep 16 18:17:24.252571 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:24.252532    2554 scope.go:117] "RemoveContainer" containerID="5bb50fab82e116d0d9c5536a5d24abf6bde7bb0ee0dd450f9595fabb6f910a3e"
Sep 16 18:17:24.252717 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:24.252692    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-bz98l_openshift-console-operator(aab214d7-cd3b-4018-b947-4e6c626eea04)\"" pod="openshift-console-operator/console-operator-696884fb9c-bz98l" podUID="aab214d7-cd3b-4018-b947-4e6c626eea04"
Sep 16 18:17:25.055837 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:25.055801    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b6hff_e1ee1605-7d05-4668-98e6-8249ac3e1052/node-ca/0.log"
Sep 16 18:17:25.258277 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:25.258221    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cvns7" event={"ID":"9f55b4e3-4cf9-4ddb-895c-5cdde11c2373","Type":"ContainerStarted","Data":"425e7f877139dd6d7d44d4aa74f72a0260cd262006f00f8f9a9bc2d5f769dc78"}
Sep 16 18:17:25.258746 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:25.258304    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cvns7" event={"ID":"9f55b4e3-4cf9-4ddb-895c-5cdde11c2373","Type":"ContainerStarted","Data":"c0bf9933c186b36bb2924694d0cd89cfd7d317b27f12510dd56e1b9482027a34"}
Sep 16 18:17:25.283938 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:25.283889    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-cvns7" podStartSLOduration=1.725948257 podStartE2EDuration="3.283870193s" totalImagesPullingTime="1.557921936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:25.283334675 +0000 UTC m=+68.842502509" watchObservedRunningTime="2026-09-16 18:17:25.283870193 +0000 UTC m=+68.843038072"
Sep 16 18:17:25.877244 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:25.877200    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:25.877433 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:25.877319    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:25.877433 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:25.877350    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:17:25.877433 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:25.877431    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm podName:f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:29.877412495 +0000 UTC m=+73.436580328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5z599" (UID: "f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1") : configmap "kube-rbac-proxy" not found
Sep 16 18:17:25.877606 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:25.877432    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:17:25.877606 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:25.877472    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls podName:f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:29.877462125 +0000 UTC m=+73.436629939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5z599" (UID: "f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:17:26.060773 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:26.060739    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zs8q4_9569d7bb-5ebe-4d49-8a92-faed5784b8c7/serve-healthcheck-canary/0.log"
Sep 16 18:17:27.266880 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:27.266849    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7w74g" event={"ID":"50b4667a-4d35-44cb-865b-747a72c7c3e6","Type":"ContainerStarted","Data":"c4d3ee7d3b3524773b2e8e3d710064b5f878742070d216caea53e7e95d8c7779"}
Sep 16 18:17:27.285322 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:27.285274    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7w74g" podStartSLOduration=65.77186506 podStartE2EDuration="1m9.285260161s" totalImagesPullingTime="3.513395101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:27.284519406 +0000 UTC m=+70.843687241" watchObservedRunningTime="2026-09-16 18:17:27.285260161 +0000 UTC m=+70.844427985"
Sep 16 18:17:27.536347 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:27.536260    2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:27.536347 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:27.536348    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:27.536668 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:27.536655    2554 scope.go:117] "RemoveContainer" containerID="5bb50fab82e116d0d9c5536a5d24abf6bde7bb0ee0dd450f9595fabb6f910a3e"
Sep 16 18:17:27.536811 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:27.536793    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-bz98l_openshift-console-operator(aab214d7-cd3b-4018-b947-4e6c626eea04)\"" pod="openshift-console-operator/console-operator-696884fb9c-bz98l" podUID="aab214d7-cd3b-4018-b947-4e6c626eea04"
Sep 16 18:17:29.907818 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:29.907778    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:29.908180 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:29.907867    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:29.908180 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:29.907939    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:17:29.908180 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:29.907964    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:17:29.908180 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:29.908007    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm podName:f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:37.907988343 +0000 UTC m=+81.467156161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5z599" (UID: "f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1") : configmap "kube-rbac-proxy" not found
Sep 16 18:17:29.908180 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:29.908021    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls podName:f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:37.908014926 +0000 UTC m=+81.467182739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5z599" (UID: "f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:17:37.968896 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:37.968859    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:37.969411 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:37.969054    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:37.969411 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:37.969169    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:17:37.969411 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:37.969224    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls podName:f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:53.969209061 +0000 UTC m=+97.528376873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5z599" (UID: "f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:17:37.969411 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:37.969396    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:38.877615 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:38.877588    2554 scope.go:117] "RemoveContainer" containerID="5bb50fab82e116d0d9c5536a5d24abf6bde7bb0ee0dd450f9595fabb6f910a3e"
Sep 16 18:17:39.305884 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:39.305857    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:17:39.306266 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:39.306252    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/1.log"
Sep 16 18:17:39.306322 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:39.306281    2554 generic.go:356] "Generic (PLEG): container finished" podID="aab214d7-cd3b-4018-b947-4e6c626eea04" containerID="ceca0b0e1b3c7ff15b3ee7991d24ea2cc682d17462e62218ac6b08dc6bbcef60" exitCode=255
Sep 16 18:17:39.306361 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:39.306343    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bz98l" event={"ID":"aab214d7-cd3b-4018-b947-4e6c626eea04","Type":"ContainerDied","Data":"ceca0b0e1b3c7ff15b3ee7991d24ea2cc682d17462e62218ac6b08dc6bbcef60"}
Sep 16 18:17:39.306400 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:39.306388    2554 scope.go:117] "RemoveContainer" containerID="5bb50fab82e116d0d9c5536a5d24abf6bde7bb0ee0dd450f9595fabb6f910a3e"
Sep 16 18:17:39.306633 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:39.306620    2554 scope.go:117] "RemoveContainer" containerID="ceca0b0e1b3c7ff15b3ee7991d24ea2cc682d17462e62218ac6b08dc6bbcef60"
Sep 16 18:17:39.306799 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:39.306781    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-bz98l_openshift-console-operator(aab214d7-cd3b-4018-b947-4e6c626eea04)\"" pod="openshift-console-operator/console-operator-696884fb9c-bz98l" podUID="aab214d7-cd3b-4018-b947-4e6c626eea04"
Sep 16 18:17:40.314455 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:40.314425    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:17:44.722326 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:44.722291    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q"]
Sep 16 18:17:44.725177 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:44.725162    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q"
Sep 16 18:17:44.727586 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:44.727562    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:17:44.728533 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:44.728519    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 18:17:44.730120 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:44.730102    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:17:44.732847 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:44.732830    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:17:44.739625 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:44.739601    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q"]
Sep 16 18:17:44.818489 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:44.818457    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6f6791c-2a9e-4301-bf0e-685b967e83e8-tmp\") pod \"klusterlet-addon-workmgr-c8b99bcc8-7qp8q\" (UID: \"e6f6791c-2a9e-4301-bf0e-685b967e83e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q"
Sep 16 18:17:44.818489 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:44.818494    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hnnk\" (UniqueName: \"kubernetes.io/projected/e6f6791c-2a9e-4301-bf0e-685b967e83e8-kube-api-access-9hnnk\") pod \"klusterlet-addon-workmgr-c8b99bcc8-7qp8q\" (UID: \"e6f6791c-2a9e-4301-bf0e-685b967e83e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q"
Sep 16 18:17:44.818722 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:44.818595    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e6f6791c-2a9e-4301-bf0e-685b967e83e8-klusterlet-config\") pod \"klusterlet-addon-workmgr-c8b99bcc8-7qp8q\" (UID: \"e6f6791c-2a9e-4301-bf0e-685b967e83e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q"
Sep 16 18:17:44.919709 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:44.919678    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e6f6791c-2a9e-4301-bf0e-685b967e83e8-klusterlet-config\") pod \"klusterlet-addon-workmgr-c8b99bcc8-7qp8q\" (UID: \"e6f6791c-2a9e-4301-bf0e-685b967e83e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q"
Sep 16 18:17:44.919709 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:44.919715    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6f6791c-2a9e-4301-bf0e-685b967e83e8-tmp\") pod \"klusterlet-addon-workmgr-c8b99bcc8-7qp8q\" (UID: \"e6f6791c-2a9e-4301-bf0e-685b967e83e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q"
Sep 16 18:17:44.919887 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:44.919732    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hnnk\" (UniqueName: \"kubernetes.io/projected/e6f6791c-2a9e-4301-bf0e-685b967e83e8-kube-api-access-9hnnk\") pod \"klusterlet-addon-workmgr-c8b99bcc8-7qp8q\" (UID: \"e6f6791c-2a9e-4301-bf0e-685b967e83e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q"
Sep 16 18:17:44.920054 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:44.920035    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6f6791c-2a9e-4301-bf0e-685b967e83e8-tmp\") pod \"klusterlet-addon-workmgr-c8b99bcc8-7qp8q\" (UID: \"e6f6791c-2a9e-4301-bf0e-685b967e83e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q"
Sep 16 18:17:44.921967 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:44.921943    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e6f6791c-2a9e-4301-bf0e-685b967e83e8-klusterlet-config\") pod \"klusterlet-addon-workmgr-c8b99bcc8-7qp8q\" (UID: \"e6f6791c-2a9e-4301-bf0e-685b967e83e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q"
Sep 16 18:17:44.931097 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:44.931074    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hnnk\" (UniqueName: \"kubernetes.io/projected/e6f6791c-2a9e-4301-bf0e-685b967e83e8-kube-api-access-9hnnk\") pod \"klusterlet-addon-workmgr-c8b99bcc8-7qp8q\" (UID: \"e6f6791c-2a9e-4301-bf0e-685b967e83e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q"
Sep 16 18:17:45.034030 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:45.034010    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q"
Sep 16 18:17:45.306016 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:45.305939    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q"]
Sep 16 18:17:45.308815 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:17:45.308781    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f6791c_2a9e_4301_bf0e_685b967e83e8.slice/crio-7ca1b30f843994865824cee7e3d0094cf5ff33a1dc860f89a86ae2f60aff153e WatchSource:0}: Error finding container 7ca1b30f843994865824cee7e3d0094cf5ff33a1dc860f89a86ae2f60aff153e: Status 404 returned error can't find the container with id 7ca1b30f843994865824cee7e3d0094cf5ff33a1dc860f89a86ae2f60aff153e
Sep 16 18:17:45.329657 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:45.329623    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q" event={"ID":"e6f6791c-2a9e-4301-bf0e-685b967e83e8","Type":"ContainerStarted","Data":"7ca1b30f843994865824cee7e3d0094cf5ff33a1dc860f89a86ae2f60aff153e"}
Sep 16 18:17:47.536097 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:47.536059    2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:47.536588 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:47.536122    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:17:47.536588 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:47.536505    2554 scope.go:117] "RemoveContainer" containerID="ceca0b0e1b3c7ff15b3ee7991d24ea2cc682d17462e62218ac6b08dc6bbcef60"
Sep 16 18:17:47.536724 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:47.536697    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-bz98l_openshift-console-operator(aab214d7-cd3b-4018-b947-4e6c626eea04)\"" pod="openshift-console-operator/console-operator-696884fb9c-bz98l" podUID="aab214d7-cd3b-4018-b947-4e6c626eea04"
Sep 16 18:17:50.347029 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:50.346996    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q" event={"ID":"e6f6791c-2a9e-4301-bf0e-685b967e83e8","Type":"ContainerStarted","Data":"85525427fff301feaf705cf875980908966b7c0bef1cc1f019031099229ae048"}
Sep 16 18:17:50.347421 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:50.347185    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q"
Sep 16 18:17:50.348751 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:50.348729    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q"
Sep 16 18:17:50.364831 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:50.364789    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c8b99bcc8-7qp8q" podStartSLOduration=1.890210685 podStartE2EDuration="6.364777133s" totalImagesPullingTime="4.474566448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:50.363625117 +0000 UTC m=+93.922792950" watchObservedRunningTime="2026-09-16 18:17:50.364777133 +0000 UTC m=+93.923944965"
Sep 16 18:17:51.876224 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:51.876188    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-l5xrt"]
Sep 16 18:17:51.878812 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:51.878793    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-l5xrt"
Sep 16 18:17:51.881403 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:51.881373    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:17:51.881403 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:51.881376    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9f6pl\""
Sep 16 18:17:51.892929 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:51.892908    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-l5xrt"]
Sep 16 18:17:51.983056 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:51.983017    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bc98c7b9-3108-4ee2-b7c4-189338bcabc3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-l5xrt\" (UID: \"bc98c7b9-3108-4ee2-b7c4-189338bcabc3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-l5xrt"
Sep 16 18:17:52.083989 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:52.083951    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bc98c7b9-3108-4ee2-b7c4-189338bcabc3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-l5xrt\" (UID: \"bc98c7b9-3108-4ee2-b7c4-189338bcabc3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-l5xrt"
Sep 16 18:17:52.084134 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:52.084098    2554 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:17:52.084181 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:52.084162    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc98c7b9-3108-4ee2-b7c4-189338bcabc3-tls-certificates podName:bc98c7b9-3108-4ee2-b7c4-189338bcabc3 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:52.584145881 +0000 UTC m=+96.143313693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/bc98c7b9-3108-4ee2-b7c4-189338bcabc3-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-l5xrt" (UID: "bc98c7b9-3108-4ee2-b7c4-189338bcabc3") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:17:52.588113 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:52.588072    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bc98c7b9-3108-4ee2-b7c4-189338bcabc3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-l5xrt\" (UID: \"bc98c7b9-3108-4ee2-b7c4-189338bcabc3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-l5xrt"
Sep 16 18:17:52.590448 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:52.590431    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bc98c7b9-3108-4ee2-b7c4-189338bcabc3-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-l5xrt\" (UID: \"bc98c7b9-3108-4ee2-b7c4-189338bcabc3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-l5xrt"
Sep 16 18:17:52.787322 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:52.787281    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-l5xrt"
Sep 16 18:17:53.053841 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:53.053810    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-l5xrt"]
Sep 16 18:17:53.056601 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:17:53.056576    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc98c7b9_3108_4ee2_b7c4_189338bcabc3.slice/crio-c6c6dbd04f8d0a99a8862946233f0f2e7a5997c73b062e354a1eef9f86c5084e WatchSource:0}: Error finding container c6c6dbd04f8d0a99a8862946233f0f2e7a5997c73b062e354a1eef9f86c5084e: Status 404 returned error can't find the container with id c6c6dbd04f8d0a99a8862946233f0f2e7a5997c73b062e354a1eef9f86c5084e
Sep 16 18:17:53.357283 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:53.357191    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-l5xrt" event={"ID":"bc98c7b9-3108-4ee2-b7c4-189338bcabc3","Type":"ContainerStarted","Data":"c6c6dbd04f8d0a99a8862946233f0f2e7a5997c73b062e354a1eef9f86c5084e"}
Sep 16 18:17:54.000092 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:54.000054    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:54.002506 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:54.002482    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5z599\" (UID: \"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1\") " pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:54.255826 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:54.255748    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b29qk"
Sep 16 18:17:54.278410 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:54.278381    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5z599"
Sep 16 18:17:54.362101 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:54.362071    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-l5xrt" event={"ID":"bc98c7b9-3108-4ee2-b7c4-189338bcabc3","Type":"ContainerStarted","Data":"8217c4b4d8a983cbe1415ed3c3d77432725d5767344aa9c7bb3e33178bbe070c"}
Sep 16 18:17:54.563754 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:54.563683    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5z599"]
Sep 16 18:17:54.566782 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:17:54.566757    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf853fe08_8ee6_4e56_92fd_6a2e9cd7a6b1.slice/crio-6e363b26e5d9b6f62cf385ed2ba4f817a0a89eb8bbfb23d0dd582b6111a1b941 WatchSource:0}: Error finding container 6e363b26e5d9b6f62cf385ed2ba4f817a0a89eb8bbfb23d0dd582b6111a1b941: Status 404 returned error can't find the container with id 6e363b26e5d9b6f62cf385ed2ba4f817a0a89eb8bbfb23d0dd582b6111a1b941
Sep 16 18:17:55.210800 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.210711    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:17:55.211002 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.210811    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:17:55.213067 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.213031    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90a6aab5-3c38-4775-ba9c-3dc99fc92207-metrics-tls\") pod \"dns-default-pbwcp\" (UID: \"90a6aab5-3c38-4775-ba9c-3dc99fc92207\") " pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:17:55.213161 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.213144    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls\") pod \"image-registry-6cfb89dbc-l9d7f\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") " pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:17:55.219439 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.219422    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f8sdc\""
Sep 16 18:17:55.227384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.227369    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:17:55.366619 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.366588    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5z599" event={"ID":"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1","Type":"ContainerStarted","Data":"c5a1b7e73f759b6960671d5fe91d17b277bae18d15366811308687ae50269fa9"}
Sep 16 18:17:55.366980 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.366644    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5z599" event={"ID":"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1","Type":"ContainerStarted","Data":"6e363b26e5d9b6f62cf385ed2ba4f817a0a89eb8bbfb23d0dd582b6111a1b941"}
Sep 16 18:17:55.366980 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.366746    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-l5xrt"
Sep 16 18:17:55.371192 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.371168    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-l5xrt"
Sep 16 18:17:55.388631 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.387183    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-l5xrt" podStartSLOduration=3.169435444 podStartE2EDuration="4.387164124s" totalImagesPullingTime="1.21772868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:55.384104753 +0000 UTC m=+98.943272586" watchObservedRunningTime="2026-09-16 18:17:55.387164124 +0000 UTC m=+98.946331957"
Sep 16 18:17:55.504040 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.504008    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mlk5x\""
Sep 16 18:17:55.505542 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.505516    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pbwcp"]
Sep 16 18:17:55.508664 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:17:55.508641    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a6aab5_3c38_4775_ba9c_3dc99fc92207.slice/crio-2e557f0c8360616ac256c98e56122af2369ce615f60bf5a5da4063ed0c88551d WatchSource:0}: Error finding container 2e557f0c8360616ac256c98e56122af2369ce615f60bf5a5da4063ed0c88551d: Status 404 returned error can't find the container with id 2e557f0c8360616ac256c98e56122af2369ce615f60bf5a5da4063ed0c88551d
Sep 16 18:17:55.512015 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.511987    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:17:55.803150 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.803083    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cfb89dbc-l9d7f"]
Sep 16 18:17:55.805781 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:17:55.805753    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9812ded9_6393_45db_9c21_338afa68c6b9.slice/crio-1bb04057928474e76b37bdb4f57c9acbe0aac97318203be76465da79a2feacd2 WatchSource:0}: Error finding container 1bb04057928474e76b37bdb4f57c9acbe0aac97318203be76465da79a2feacd2: Status 404 returned error can't find the container with id 1bb04057928474e76b37bdb4f57c9acbe0aac97318203be76465da79a2feacd2
Sep 16 18:17:55.932486 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.932450    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"]
Sep 16 18:17:55.950370 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.950333    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"]
Sep 16 18:17:55.950525 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.950464    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"
Sep 16 18:17:55.953184 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.953165    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:17:55.954022 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.954003    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:17:55.954128 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.954027    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-p797m\""
Sep 16 18:17:55.954128 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.954003    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 18:17:55.954128 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.954071    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 18:17:55.954128 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:55.954071    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:17:56.118468 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.118423    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bf91448-f095-489f-813b-0a290a5ff26b-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-pd59k\" (UID: \"0bf91448-f095-489f-813b-0a290a5ff26b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"
Sep 16 18:17:56.118660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.118546    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0bf91448-f095-489f-813b-0a290a5ff26b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-pd59k\" (UID: \"0bf91448-f095-489f-813b-0a290a5ff26b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"
Sep 16 18:17:56.118660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.118588    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0bf91448-f095-489f-813b-0a290a5ff26b-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-pd59k\" (UID: \"0bf91448-f095-489f-813b-0a290a5ff26b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"
Sep 16 18:17:56.118660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.118628    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqlt5\" (UniqueName: \"kubernetes.io/projected/0bf91448-f095-489f-813b-0a290a5ff26b-kube-api-access-xqlt5\") pod \"prometheus-operator-68bcd7f987-pd59k\" (UID: \"0bf91448-f095-489f-813b-0a290a5ff26b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"
Sep 16 18:17:56.219574 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.219516    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0bf91448-f095-489f-813b-0a290a5ff26b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-pd59k\" (UID: \"0bf91448-f095-489f-813b-0a290a5ff26b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"
Sep 16 18:17:56.219574 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.219576    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0bf91448-f095-489f-813b-0a290a5ff26b-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-pd59k\" (UID: \"0bf91448-f095-489f-813b-0a290a5ff26b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"
Sep 16 18:17:56.219871 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.219613    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqlt5\" (UniqueName: \"kubernetes.io/projected/0bf91448-f095-489f-813b-0a290a5ff26b-kube-api-access-xqlt5\") pod \"prometheus-operator-68bcd7f987-pd59k\" (UID: \"0bf91448-f095-489f-813b-0a290a5ff26b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"
Sep 16 18:17:56.219871 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.219664    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bf91448-f095-489f-813b-0a290a5ff26b-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-pd59k\" (UID: \"0bf91448-f095-489f-813b-0a290a5ff26b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"
Sep 16 18:17:56.219871 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:56.219814    2554 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 16 18:17:56.220017 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:56.219875    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bf91448-f095-489f-813b-0a290a5ff26b-prometheus-operator-tls podName:0bf91448-f095-489f-813b-0a290a5ff26b nodeName:}" failed. No retries permitted until 2026-09-16 18:17:56.719857136 +0000 UTC m=+100.279024964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/0bf91448-f095-489f-813b-0a290a5ff26b-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-pd59k" (UID: "0bf91448-f095-489f-813b-0a290a5ff26b") : secret "prometheus-operator-tls" not found
Sep 16 18:17:56.220486 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.220462    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0bf91448-f095-489f-813b-0a290a5ff26b-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-pd59k\" (UID: \"0bf91448-f095-489f-813b-0a290a5ff26b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"
Sep 16 18:17:56.222437 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.222411    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0bf91448-f095-489f-813b-0a290a5ff26b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-pd59k\" (UID: \"0bf91448-f095-489f-813b-0a290a5ff26b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"
Sep 16 18:17:56.231773 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.231744    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqlt5\" (UniqueName: \"kubernetes.io/projected/0bf91448-f095-489f-813b-0a290a5ff26b-kube-api-access-xqlt5\") pod \"prometheus-operator-68bcd7f987-pd59k\" (UID: \"0bf91448-f095-489f-813b-0a290a5ff26b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"
Sep 16 18:17:56.372620 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.372547    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f" event={"ID":"9812ded9-6393-45db-9c21-338afa68c6b9","Type":"ContainerStarted","Data":"41e7d35dfdcae674041139ee119cbb4d317e4e2a359ad7c7b25f2f0204cfe1ec"}
Sep 16 18:17:56.372620 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.372596    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f" event={"ID":"9812ded9-6393-45db-9c21-338afa68c6b9","Type":"ContainerStarted","Data":"1bb04057928474e76b37bdb4f57c9acbe0aac97318203be76465da79a2feacd2"}
Sep 16 18:17:56.373123 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.372642    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:17:56.373988 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.373965    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pbwcp" event={"ID":"90a6aab5-3c38-4775-ba9c-3dc99fc92207","Type":"ContainerStarted","Data":"2e557f0c8360616ac256c98e56122af2369ce615f60bf5a5da4063ed0c88551d"}
Sep 16 18:17:56.375929 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.375901    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5z599" event={"ID":"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1","Type":"ContainerStarted","Data":"4b70449dcb80642c1184f1bdeffb9a21caa96462fe229d05c65c182772611423"}
Sep 16 18:17:56.394294 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.394249    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f" podStartSLOduration=98.394212365 podStartE2EDuration="1m38.394212365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:56.393037324 +0000 UTC m=+99.952205156" watchObservedRunningTime="2026-09-16 18:17:56.394212365 +0000 UTC m=+99.953380200"
Sep 16 18:17:56.724617 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.724533    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bf91448-f095-489f-813b-0a290a5ff26b-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-pd59k\" (UID: \"0bf91448-f095-489f-813b-0a290a5ff26b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"
Sep 16 18:17:56.727710 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.727680    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bf91448-f095-489f-813b-0a290a5ff26b-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-pd59k\" (UID: \"0bf91448-f095-489f-813b-0a290a5ff26b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"
Sep 16 18:17:56.859960 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:56.859926    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"
Sep 16 18:17:58.298029 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:58.298000    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-pd59k"]
Sep 16 18:17:58.301180 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:17:58.301157    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf91448_f095_489f_813b_0a290a5ff26b.slice/crio-0e04df087eea7d4575fea1ba3e718a10ce47b6867979ccd904b3325b67073fc4 WatchSource:0}: Error finding container 0e04df087eea7d4575fea1ba3e718a10ce47b6867979ccd904b3325b67073fc4: Status 404 returned error can't find the container with id 0e04df087eea7d4575fea1ba3e718a10ce47b6867979ccd904b3325b67073fc4
Sep 16 18:17:58.384333 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:58.384297    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5z599" event={"ID":"f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1","Type":"ContainerStarted","Data":"ccc63fbad6a2b683b71d478c8c80cbf7e4da5c3b73142577a0ab4ba042d63d57"}
Sep 16 18:17:58.385456 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:58.385433    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k" event={"ID":"0bf91448-f095-489f-813b-0a290a5ff26b","Type":"ContainerStarted","Data":"0e04df087eea7d4575fea1ba3e718a10ce47b6867979ccd904b3325b67073fc4"}
Sep 16 18:17:58.387029 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:58.387008    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pbwcp" event={"ID":"90a6aab5-3c38-4775-ba9c-3dc99fc92207","Type":"ContainerStarted","Data":"0b75edb6a2d0cd88d15bd4b5a00556cd567105be431b46d72e0d044ac86a5866"}
Sep 16 18:17:58.387106 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:58.387051    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pbwcp" event={"ID":"90a6aab5-3c38-4775-ba9c-3dc99fc92207","Type":"ContainerStarted","Data":"a5ef9ebe1ad91804114f67f1acffe9d8c72e5781daa2edd4861a68a633a668a1"}
Sep 16 18:17:58.387176 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:58.387159    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:17:58.405324 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:58.405280    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5z599" podStartSLOduration=33.160832545 podStartE2EDuration="36.40526465s" totalImagesPullingTime="3.244432105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:17:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:58.403988211 +0000 UTC m=+101.963156068" watchObservedRunningTime="2026-09-16 18:17:58.40526465 +0000 UTC m=+101.964432490"
Sep 16 18:17:58.425512 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:58.425477    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pbwcp" podStartSLOduration=64.92252663 podStartE2EDuration="1m7.425467544s" totalImagesPullingTime="2.502940914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:58.424752661 +0000 UTC m=+101.983920493" watchObservedRunningTime="2026-09-16 18:17:58.425467544 +0000 UTC m=+101.984635375"
Sep 16 18:17:58.882433 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:17:58.882410    2554 scope.go:117] "RemoveContainer" containerID="ceca0b0e1b3c7ff15b3ee7991d24ea2cc682d17462e62218ac6b08dc6bbcef60"
Sep 16 18:17:58.882578 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:17:58.882547    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-bz98l_openshift-console-operator(aab214d7-cd3b-4018-b947-4e6c626eea04)\"" pod="openshift-console-operator/console-operator-696884fb9c-bz98l" podUID="aab214d7-cd3b-4018-b947-4e6c626eea04"
Sep 16 18:18:00.396666 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:00.396617    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k" event={"ID":"0bf91448-f095-489f-813b-0a290a5ff26b","Type":"ContainerStarted","Data":"4a5db970f0ec6de646bbed6b1798800932cc2902c16913a4578779fa418b95d8"}
Sep 16 18:18:00.397115 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:00.396680    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k" event={"ID":"0bf91448-f095-489f-813b-0a290a5ff26b","Type":"ContainerStarted","Data":"355946f9129f9b3f77951791ca957cbc63125e838576807a59e8393170466e1e"}
Sep 16 18:18:00.415337 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:00.415288    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pd59k" podStartSLOduration=3.871158759 podStartE2EDuration="5.41527135s" totalImagesPullingTime="1.544112591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:18:00.414841877 +0000 UTC m=+103.974009710" watchObservedRunningTime="2026-09-16 18:18:00.41527135 +0000 UTC m=+103.974439182"
Sep 16 18:18:02.326360 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.326324    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rvf8s"]
Sep 16 18:18:02.329031 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.329010    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.331717 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.331694    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:18:02.331831 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.331811    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-975dg\""
Sep 16 18:18:02.332047 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.332030    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:18:02.332119 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.332051    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:18:02.468511 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.468477    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvdjs\" (UniqueName: \"kubernetes.io/projected/7b64bf66-02e1-4598-8815-161511160ff4-kube-api-access-hvdjs\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.468679 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.468518    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-textfile\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.468679 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.468543    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b64bf66-02e1-4598-8815-161511160ff4-metrics-client-ca\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.468679 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.468567    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b64bf66-02e1-4598-8815-161511160ff4-root\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.468679 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.468615    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b64bf66-02e1-4598-8815-161511160ff4-sys\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.468679 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.468658    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.468833 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.468682    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.468833 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.468731    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-wtmp\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.468833 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.468756    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-tls\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.569702 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.569665    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-tls\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.569888 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.569721    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdjs\" (UniqueName: \"kubernetes.io/projected/7b64bf66-02e1-4598-8815-161511160ff4-kube-api-access-hvdjs\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.569888 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.569747    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-textfile\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.569888 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.569765    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b64bf66-02e1-4598-8815-161511160ff4-metrics-client-ca\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.569888 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.569786    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b64bf66-02e1-4598-8815-161511160ff4-root\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.569888 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.569812    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b64bf66-02e1-4598-8815-161511160ff4-sys\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.569888 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.569839    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.569888 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:18:02.569841    2554 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:18:02.570187 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.569893    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b64bf66-02e1-4598-8815-161511160ff4-sys\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.570187 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:18:02.569922    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-tls podName:7b64bf66-02e1-4598-8815-161511160ff4 nodeName:}" failed. No retries permitted until 2026-09-16 18:18:03.06990111 +0000 UTC m=+106.629068938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-tls") pod "node-exporter-rvf8s" (UID: "7b64bf66-02e1-4598-8815-161511160ff4") : secret "node-exporter-tls" not found
Sep 16 18:18:02.570187 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.569940    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.570187 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.569983    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-wtmp\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.570187 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.569896    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b64bf66-02e1-4598-8815-161511160ff4-root\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.570187 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.570133    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-wtmp\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.570187 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.570162    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-textfile\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.570502 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.570480    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b64bf66-02e1-4598-8815-161511160ff4-metrics-client-ca\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.570536 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.570480    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.572077 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.572045    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:02.578205 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:02.578158    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvdjs\" (UniqueName: \"kubernetes.io/projected/7b64bf66-02e1-4598-8815-161511160ff4-kube-api-access-hvdjs\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:03.074372 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.074334    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-tls\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:03.076546 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.076527    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b64bf66-02e1-4598-8815-161511160ff4-node-exporter-tls\") pod \"node-exporter-rvf8s\" (UID: \"7b64bf66-02e1-4598-8815-161511160ff4\") " pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:03.237443 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.237411    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rvf8s"
Sep 16 18:18:03.247985 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:18:03.247952    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b64bf66_02e1_4598_8815_161511160ff4.slice/crio-e91a801812715a0023a201fab2a7ca119966c2d3508961b1bf1e0e7b839f02b2 WatchSource:0}: Error finding container e91a801812715a0023a201fab2a7ca119966c2d3508961b1bf1e0e7b839f02b2: Status 404 returned error can't find the container with id e91a801812715a0023a201fab2a7ca119966c2d3508961b1bf1e0e7b839f02b2
Sep 16 18:18:03.382733 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.382655    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:18:03.385728 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.385708    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.388275 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.388046    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:18:03.388395 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.388355    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:18:03.388453 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.388410    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:18:03.388502 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.388470    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:18:03.388502 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.388482    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:18:03.388781 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.388761    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sn2np\""
Sep 16 18:18:03.388880 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.388837    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:18:03.388941 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.388878    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:18:03.389146 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.389128    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:18:03.395744 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.395725    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:18:03.403457 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.403440    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:18:03.409943 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.409914    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvf8s" event={"ID":"7b64bf66-02e1-4598-8815-161511160ff4","Type":"ContainerStarted","Data":"e91a801812715a0023a201fab2a7ca119966c2d3508961b1bf1e0e7b839f02b2"}
Sep 16 18:18:03.477606 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.477577    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.477788 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.477611    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.477788 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.477634    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.477788 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.477660    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f02331d3-c795-4f3b-9949-20b462c55065-config-out\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.477788 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.477725    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f02331d3-c795-4f3b-9949-20b462c55065-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.477788 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.477772    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpn76\" (UniqueName: \"kubernetes.io/projected/f02331d3-c795-4f3b-9949-20b462c55065-kube-api-access-fpn76\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.477987 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.477795    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f02331d3-c795-4f3b-9949-20b462c55065-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.477987 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.477819    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.477987 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.477905    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-config-volume\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.477987 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.477938    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f02331d3-c795-4f3b-9949-20b462c55065-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.477987 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.477959    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f02331d3-c795-4f3b-9949-20b462c55065-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.478149 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.477987    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.478149 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.478035    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-web-config\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.578962 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.578926    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f02331d3-c795-4f3b-9949-20b462c55065-config-out\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.578962 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.578961    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f02331d3-c795-4f3b-9949-20b462c55065-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.579211 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.578979    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpn76\" (UniqueName: \"kubernetes.io/projected/f02331d3-c795-4f3b-9949-20b462c55065-kube-api-access-fpn76\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.579211 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.578996    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f02331d3-c795-4f3b-9949-20b462c55065-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.579211 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.579010    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.579211 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.579048    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-config-volume\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.579211 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.579078    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f02331d3-c795-4f3b-9949-20b462c55065-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.579211 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:18:03.579143    2554 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 16 18:18:03.579211 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.579188    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f02331d3-c795-4f3b-9949-20b462c55065-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.579211 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:18:03.579201    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-main-tls podName:f02331d3-c795-4f3b-9949-20b462c55065 nodeName:}" failed. No retries permitted until 2026-09-16 18:18:04.079183614 +0000 UTC m=+107.638351431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "f02331d3-c795-4f3b-9949-20b462c55065") : secret "alertmanager-main-tls" not found
Sep 16 18:18:03.579671 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.579272    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.579671 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.579313    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-web-config\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.579671 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.579366    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.579671 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.579397    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.579671 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.579434    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.579671 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.579528    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f02331d3-c795-4f3b-9949-20b462c55065-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.580004 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.579975    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f02331d3-c795-4f3b-9949-20b462c55065-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.580956 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.580935    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f02331d3-c795-4f3b-9949-20b462c55065-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.581795 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.581772    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f02331d3-c795-4f3b-9949-20b462c55065-config-out\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.582369 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.582347    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.582463 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.582442    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-config-volume\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.582519 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.582465    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f02331d3-c795-4f3b-9949-20b462c55065-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.582519 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.582481    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.582618 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.582593    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.582737 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.582720    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-web-config\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.583250 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.583218    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:03.588397 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:03.588377    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpn76\" (UniqueName: \"kubernetes.io/projected/f02331d3-c795-4f3b-9949-20b462c55065-kube-api-access-fpn76\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:04.083575 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:04.083528    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:04.086166 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:04.086132    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:04.296620 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:04.296591    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:18:04.415105 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:04.415075    2554 generic.go:356] "Generic (PLEG): container finished" podID="7b64bf66-02e1-4598-8815-161511160ff4" containerID="48572f4bba246cda76006dee04d08375dff065aa532dfdd2de585b259a779603" exitCode=0
Sep 16 18:18:04.415472 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:04.415130    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvf8s" event={"ID":"7b64bf66-02e1-4598-8815-161511160ff4","Type":"ContainerDied","Data":"48572f4bba246cda76006dee04d08375dff065aa532dfdd2de585b259a779603"}
Sep 16 18:18:04.573176 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:04.573146    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:18:04.576067 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:18:04.576044    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02331d3_c795_4f3b_9949_20b462c55065.slice/crio-9747b9ad76294f3e87d85f719994a01f111072aad30045992ec277e3f494e533 WatchSource:0}: Error finding container 9747b9ad76294f3e87d85f719994a01f111072aad30045992ec277e3f494e533: Status 404 returned error can't find the container with id 9747b9ad76294f3e87d85f719994a01f111072aad30045992ec277e3f494e533
Sep 16 18:18:05.421524 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:05.421480    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvf8s" event={"ID":"7b64bf66-02e1-4598-8815-161511160ff4","Type":"ContainerStarted","Data":"ac56d093f5edfaabcfed0b08f7828609a6780a7043ec401ef8271f5802437d12"}
Sep 16 18:18:05.421967 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:05.421556    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvf8s" event={"ID":"7b64bf66-02e1-4598-8815-161511160ff4","Type":"ContainerStarted","Data":"4546d6954f512259d766fbf8afb8ceefceed795ffef75e2fe7b56b77edf01e56"}
Sep 16 18:18:05.422830 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:05.422807    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f02331d3-c795-4f3b-9949-20b462c55065","Type":"ContainerStarted","Data":"9747b9ad76294f3e87d85f719994a01f111072aad30045992ec277e3f494e533"}
Sep 16 18:18:05.444885 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:05.444806    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rvf8s" podStartSLOduration=2.53032703 podStartE2EDuration="3.444789205s" totalImagesPullingTime="853.605764ms" totalInitContainerRuntime="60.856411ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:18:05.444145463 +0000 UTC m=+109.003313295" watchObservedRunningTime="2026-09-16 18:18:05.444789205 +0000 UTC m=+109.003957034"
Sep 16 18:18:06.427556 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:06.427526    2554 generic.go:356] "Generic (PLEG): container finished" podID="f02331d3-c795-4f3b-9949-20b462c55065" containerID="9384ccfe408a7e5ee2642207045a1aab25c932341437fc4117a8376f9f75e7be" exitCode=0
Sep 16 18:18:06.427947 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:06.427618    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f02331d3-c795-4f3b-9949-20b462c55065","Type":"ContainerDied","Data":"9384ccfe408a7e5ee2642207045a1aab25c932341437fc4117a8376f9f75e7be"}
Sep 16 18:18:08.393793 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.393706    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pbwcp"
Sep 16 18:18:08.439750 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.439715    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f02331d3-c795-4f3b-9949-20b462c55065","Type":"ContainerStarted","Data":"0987ea6e18c2200161c2d8d12c0674402e30432244a58da9f7f07bb37fe68120"}
Sep 16 18:18:08.439916 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.439798    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f02331d3-c795-4f3b-9949-20b462c55065","Type":"ContainerStarted","Data":"7500f371705e62af9f0ba972936a24d885081ba01502c9ddebc9255dd71efe36"}
Sep 16 18:18:08.439916 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.439835    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f02331d3-c795-4f3b-9949-20b462c55065","Type":"ContainerStarted","Data":"c587e0e10955c4737dd0c3ff6a7ea7049e4b82c6d609c3f104d996d08a119e71"}
Sep 16 18:18:08.439916 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.439870    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f02331d3-c795-4f3b-9949-20b462c55065","Type":"ContainerStarted","Data":"2af5cdb3cc8e64b91166a74031e6837cb19194d0755ffb5ade43a85cb72041da"}
Sep 16 18:18:08.439916 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.439906    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f02331d3-c795-4f3b-9949-20b462c55065","Type":"ContainerStarted","Data":"7078849921a9dfad0f6e4879a25af35169955bf3f287d387c1d3c822788ef3bb"}
Sep 16 18:18:08.531220 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.531180    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:18:08.534102 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.534079    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.536533 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.536482    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:18:08.536533 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.536485    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:18:08.537756 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.537732    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:18:08.537983 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.537964    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:18:08.538080 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.538035    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:18:08.538080 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.538050    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:18:08.538189 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.538078    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:18:08.538189 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.538088    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:18:08.538189 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.538160    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:18:08.538430 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.538409    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:18:08.538534 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.538441    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:18:08.538601 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.538535    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bws42\""
Sep 16 18:18:08.538651 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.538621    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:18:08.538703 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.538672    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bmpohrcrmq569\""
Sep 16 18:18:08.538703 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.538624    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:18:08.539001 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.538985    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:18:08.541578 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.541560    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:18:08.557213 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.557178    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:18:08.723121 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723038    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723121 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723079    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723121 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723102    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddeab720-40e2-43e3-be8c-c6457f48905d-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723400 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723141    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723400 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723187    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-config\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723400 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723262    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723400 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723310    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723400 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723339    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723400 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723363    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddeab720-40e2-43e3-be8c-c6457f48905d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723675 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723422    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723675 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723446    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723675 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723471    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723675 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723505    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723675 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723542    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723675 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723565    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9dst\" (UniqueName: \"kubernetes.io/projected/ddeab720-40e2-43e3-be8c-c6457f48905d-kube-api-access-x9dst\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723675 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723606    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723675 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723634    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.723675 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723660    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.724006 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723682    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.724006 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.723707    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.826582    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.826632    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-config\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.826665    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.826701    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.826728    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.826752    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddeab720-40e2-43e3-be8c-c6457f48905d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.826806    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.826830    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.826857    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.826880    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.826911    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.826956    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9dst\" (UniqueName: \"kubernetes.io/projected/ddeab720-40e2-43e3-be8c-c6457f48905d-kube-api-access-x9dst\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.826993    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.827029    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.827057    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.827080    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.827374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.827106    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.828308 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.827142    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.828308 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.827194    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.828308 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.827246    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddeab720-40e2-43e3-be8c-c6457f48905d-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.828308 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.828113    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.828308 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.828153    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.828550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.828357    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.830265 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.828761    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.830747 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.830718    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.836425 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.836393    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.838191 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.838162    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.838470 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.838444    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddeab720-40e2-43e3-be8c-c6457f48905d-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.838562 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.838545    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.838624 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.838564    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.838624 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.838571    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.838726 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.838711    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.838843 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.838771    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.839625 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.839599    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.840220 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.840195    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.844287 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.844044    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddeab720-40e2-43e3-be8c-c6457f48905d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.844287 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.844224    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-config\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.844589 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.844569    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.845120 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.845089    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:08.848051 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:08.848030    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9dst\" (UniqueName: \"kubernetes.io/projected/ddeab720-40e2-43e3-be8c-c6457f48905d-kube-api-access-x9dst\") pod \"prometheus-k8s-0\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:09.146146 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:09.146112    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:09.429660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:09.429620    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:18:09.432194 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:18:09.432161    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddeab720_40e2_43e3_be8c_c6457f48905d.slice/crio-e3de0265243e0b8674c42e4927ae8996d7b49eba6082d3ed27f1c8e4b72955bb WatchSource:0}: Error finding container e3de0265243e0b8674c42e4927ae8996d7b49eba6082d3ed27f1c8e4b72955bb: Status 404 returned error can't find the container with id e3de0265243e0b8674c42e4927ae8996d7b49eba6082d3ed27f1c8e4b72955bb
Sep 16 18:18:09.446292 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:09.446269    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddeab720-40e2-43e3-be8c-c6457f48905d","Type":"ContainerStarted","Data":"e3de0265243e0b8674c42e4927ae8996d7b49eba6082d3ed27f1c8e4b72955bb"}
Sep 16 18:18:09.449729 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:09.449705    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f02331d3-c795-4f3b-9949-20b462c55065","Type":"ContainerStarted","Data":"a95bd5b7242d6867b7272c0933e35d8fdc0519e3e5c5ede44f1631b7acc9e974"}
Sep 16 18:18:09.479159 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:09.479113    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.342692787 podStartE2EDuration="6.479096427s" totalImagesPullingTime="3.121303645s" totalInitContainerRuntime="15.099995ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:18:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:18:09.477305314 +0000 UTC m=+113.036473145" watchObservedRunningTime="2026-09-16 18:18:09.479096427 +0000 UTC m=+113.038264262"
Sep 16 18:18:10.455214 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:10.455180    2554 generic.go:356] "Generic (PLEG): container finished" podID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerID="bdb3cf80bc15aefd6f288b83955552055edd867427882721f807400f600cace6" exitCode=0
Sep 16 18:18:10.455657 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:10.455273    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddeab720-40e2-43e3-be8c-c6457f48905d","Type":"ContainerDied","Data":"bdb3cf80bc15aefd6f288b83955552055edd867427882721f807400f600cace6"}
Sep 16 18:18:12.878391 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:12.878358    2554 scope.go:117] "RemoveContainer" containerID="ceca0b0e1b3c7ff15b3ee7991d24ea2cc682d17462e62218ac6b08dc6bbcef60"
Sep 16 18:18:13.470857 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:13.470825    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:18:13.471042 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:13.470922    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bz98l" event={"ID":"aab214d7-cd3b-4018-b947-4e6c626eea04","Type":"ContainerStarted","Data":"f12c0655216f6b7b18c3da91067aa76a3c669c589305bbffdee665bcf52aa8cd"}
Sep 16 18:18:13.472714 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:13.471918    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:18:13.475489 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:13.475460    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddeab720-40e2-43e3-be8c-c6457f48905d","Type":"ContainerStarted","Data":"e88247dc0b010bf07302d88598d555c1115d64b3e5cc458a3d1a1864cd811c6d"}
Sep 16 18:18:13.475584 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:13.475536    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddeab720-40e2-43e3-be8c-c6457f48905d","Type":"ContainerStarted","Data":"c656937d4db909565db02fa8dcc7e618e18d6ae9ebe0fa2c26589baaf10a7a2c"}
Sep 16 18:18:13.711724 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:13.711618    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-bz98l" podStartSLOduration=53.301623837 podStartE2EDuration="56.711599982s" totalImagesPullingTime="3.409976145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:18:13.71066313 +0000 UTC m=+117.269831199" watchObservedRunningTime="2026-09-16 18:18:13.711599982 +0000 UTC m=+117.270767816"
Sep 16 18:18:14.223425 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:14.223397    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-bz98l"
Sep 16 18:18:15.516959 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:15.516926    2554 patch_prober.go:28] interesting pod/image-registry-6cfb89dbc-l9d7f container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:18:15.517368 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:15.516976    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f" podUID="9812ded9-6393-45db-9c21-338afa68c6b9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:18:16.491832 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:16.491794    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddeab720-40e2-43e3-be8c-c6457f48905d","Type":"ContainerStarted","Data":"30dadc025eb64419180e1f17723378f692715dab4df203f05d5adff2c34a28c8"}
Sep 16 18:18:16.491993 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:16.491907    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddeab720-40e2-43e3-be8c-c6457f48905d","Type":"ContainerStarted","Data":"3979ad91ecccdd2e49c70bff4ee9dec99fe0b0fcd05541dae39efa95efcbf944"}
Sep 16 18:18:16.491993 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:16.491954    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddeab720-40e2-43e3-be8c-c6457f48905d","Type":"ContainerStarted","Data":"2b955ec62ac652801e15d150c79b3064531be65efe77dc0e62d909b657215305"}
Sep 16 18:18:16.491993 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:16.491989    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddeab720-40e2-43e3-be8c-c6457f48905d","Type":"ContainerStarted","Data":"afe7ad10cd143b4fd4ee2eb00c26fec7e6e0ed55bfb2c45e3342a847b287e2a8"}
Sep 16 18:18:16.530334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:16.530268    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.522908473 podStartE2EDuration="8.530247984s" totalImagesPullingTime="4.989851816s" totalInitContainerRuntime="17.487695ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:18:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:18:16.528096205 +0000 UTC m=+120.087264048" watchObservedRunningTime="2026-09-16 18:18:16.530247984 +0000 UTC m=+120.089415820"
Sep 16 18:18:16.767224 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:16.767138    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cfb89dbc-l9d7f"]
Sep 16 18:18:16.771765 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:16.771736    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:18:19.146345 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:19.146312    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:18:26.678617 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:26.678581    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs\") pod \"network-metrics-daemon-vmvpp\" (UID: \"e4c7d4e4-64cb-4cdd-90ae-db41d1889099\") " pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:18:26.680805 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:26.680776    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c7d4e4-64cb-4cdd-90ae-db41d1889099-metrics-certs\") pod \"network-metrics-daemon-vmvpp\" (UID: \"e4c7d4e4-64cb-4cdd-90ae-db41d1889099\") " pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:18:26.808299 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:26.808275    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5rgdl\""
Sep 16 18:18:26.816067 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:26.816048    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmvpp"
Sep 16 18:18:27.086079 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:27.086049    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vmvpp"]
Sep 16 18:18:27.089190 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:18:27.089166    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c7d4e4_64cb_4cdd_90ae_db41d1889099.slice/crio-6eb48353b55777c61a23ad7d2008e1dca70349025294a1ee1599fd16916f22cf WatchSource:0}: Error finding container 6eb48353b55777c61a23ad7d2008e1dca70349025294a1ee1599fd16916f22cf: Status 404 returned error can't find the container with id 6eb48353b55777c61a23ad7d2008e1dca70349025294a1ee1599fd16916f22cf
Sep 16 18:18:27.534150 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:27.534116    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vmvpp" event={"ID":"e4c7d4e4-64cb-4cdd-90ae-db41d1889099","Type":"ContainerStarted","Data":"6eb48353b55777c61a23ad7d2008e1dca70349025294a1ee1599fd16916f22cf"}
Sep 16 18:18:28.539054 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:28.539024    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vmvpp" event={"ID":"e4c7d4e4-64cb-4cdd-90ae-db41d1889099","Type":"ContainerStarted","Data":"7748ed65748554b2d2137192386a7dd41dc0b4937c2f95da77f012440220244e"}
Sep 16 18:18:28.539389 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:28.539074    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vmvpp" event={"ID":"e4c7d4e4-64cb-4cdd-90ae-db41d1889099","Type":"ContainerStarted","Data":"eb02ecc8e6588629f7946f88058a409f3fb7a97f79415af196dc44f8a367ec3a"}
Sep 16 18:18:28.559131 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:28.559088    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vmvpp" podStartSLOduration=130.312552865 podStartE2EDuration="2m11.559074768s" totalImagesPullingTime="1.246521903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:18:28.558417766 +0000 UTC m=+132.117585599" watchObservedRunningTime="2026-09-16 18:18:28.559074768 +0000 UTC m=+132.118242600"
Sep 16 18:18:41.788595 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:41.788558    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f" podUID="9812ded9-6393-45db-9c21-338afa68c6b9" containerName="registry" containerID="cri-o://41e7d35dfdcae674041139ee119cbb4d317e4e2a359ad7c7b25f2f0204cfe1ec" gracePeriod=30
Sep 16 18:18:42.026065 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.026035    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:18:42.091541 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.091482    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls\") pod \"9812ded9-6393-45db-9c21-338afa68c6b9\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") "
Sep 16 18:18:42.091541 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.091521    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9812ded9-6393-45db-9c21-338afa68c6b9-registry-certificates\") pod \"9812ded9-6393-45db-9c21-338afa68c6b9\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") "
Sep 16 18:18:42.091693 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.091559    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-bound-sa-token\") pod \"9812ded9-6393-45db-9c21-338afa68c6b9\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") "
Sep 16 18:18:42.091693 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.091585    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9812ded9-6393-45db-9c21-338afa68c6b9-image-registry-private-configuration\") pod \"9812ded9-6393-45db-9c21-338afa68c6b9\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") "
Sep 16 18:18:42.091693 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.091608    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9812ded9-6393-45db-9c21-338afa68c6b9-ca-trust-extracted\") pod \"9812ded9-6393-45db-9c21-338afa68c6b9\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") "
Sep 16 18:18:42.091693 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.091642    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9812ded9-6393-45db-9c21-338afa68c6b9-trusted-ca\") pod \"9812ded9-6393-45db-9c21-338afa68c6b9\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") "
Sep 16 18:18:42.091873 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.091700    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmntm\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-kube-api-access-rmntm\") pod \"9812ded9-6393-45db-9c21-338afa68c6b9\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") "
Sep 16 18:18:42.091873 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.091748    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9812ded9-6393-45db-9c21-338afa68c6b9-installation-pull-secrets\") pod \"9812ded9-6393-45db-9c21-338afa68c6b9\" (UID: \"9812ded9-6393-45db-9c21-338afa68c6b9\") "
Sep 16 18:18:42.092095 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.092065    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9812ded9-6393-45db-9c21-338afa68c6b9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9812ded9-6393-45db-9c21-338afa68c6b9" (UID: "9812ded9-6393-45db-9c21-338afa68c6b9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:18:42.092175 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.092148    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9812ded9-6393-45db-9c21-338afa68c6b9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9812ded9-6393-45db-9c21-338afa68c6b9" (UID: "9812ded9-6393-45db-9c21-338afa68c6b9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:18:42.093991 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.093966    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9812ded9-6393-45db-9c21-338afa68c6b9" (UID: "9812ded9-6393-45db-9c21-338afa68c6b9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:18:42.094159 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.094126    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-kube-api-access-rmntm" (OuterVolumeSpecName: "kube-api-access-rmntm") pod "9812ded9-6393-45db-9c21-338afa68c6b9" (UID: "9812ded9-6393-45db-9c21-338afa68c6b9"). InnerVolumeSpecName "kube-api-access-rmntm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:18:42.094288 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.094179    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9812ded9-6393-45db-9c21-338afa68c6b9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9812ded9-6393-45db-9c21-338afa68c6b9" (UID: "9812ded9-6393-45db-9c21-338afa68c6b9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:18:42.094288 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.094201    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9812ded9-6393-45db-9c21-338afa68c6b9" (UID: "9812ded9-6393-45db-9c21-338afa68c6b9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:18:42.094398 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.094352    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9812ded9-6393-45db-9c21-338afa68c6b9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9812ded9-6393-45db-9c21-338afa68c6b9" (UID: "9812ded9-6393-45db-9c21-338afa68c6b9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:18:42.099872 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.099847    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9812ded9-6393-45db-9c21-338afa68c6b9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9812ded9-6393-45db-9c21-338afa68c6b9" (UID: "9812ded9-6393-45db-9c21-338afa68c6b9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:18:42.192470 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.192448    2554 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-bound-sa-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:18:42.192470 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.192469    2554 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9812ded9-6393-45db-9c21-338afa68c6b9-image-registry-private-configuration\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:18:42.192581 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.192479    2554 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9812ded9-6393-45db-9c21-338afa68c6b9-ca-trust-extracted\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:18:42.192581 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.192488    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9812ded9-6393-45db-9c21-338afa68c6b9-trusted-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:18:42.192581 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.192496    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmntm\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-kube-api-access-rmntm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:18:42.192581 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.192504    2554 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9812ded9-6393-45db-9c21-338afa68c6b9-installation-pull-secrets\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:18:42.192581 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.192512    2554 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9812ded9-6393-45db-9c21-338afa68c6b9-registry-tls\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:18:42.192581 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.192520    2554 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9812ded9-6393-45db-9c21-338afa68c6b9-registry-certificates\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:18:42.590712 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.590676    2554 generic.go:356] "Generic (PLEG): container finished" podID="9812ded9-6393-45db-9c21-338afa68c6b9" containerID="41e7d35dfdcae674041139ee119cbb4d317e4e2a359ad7c7b25f2f0204cfe1ec" exitCode=0
Sep 16 18:18:42.590907 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.590720    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f"
Sep 16 18:18:42.590907 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.590766    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f" event={"ID":"9812ded9-6393-45db-9c21-338afa68c6b9","Type":"ContainerDied","Data":"41e7d35dfdcae674041139ee119cbb4d317e4e2a359ad7c7b25f2f0204cfe1ec"}
Sep 16 18:18:42.590907 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.590820    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cfb89dbc-l9d7f" event={"ID":"9812ded9-6393-45db-9c21-338afa68c6b9","Type":"ContainerDied","Data":"1bb04057928474e76b37bdb4f57c9acbe0aac97318203be76465da79a2feacd2"}
Sep 16 18:18:42.590907 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.590847    2554 scope.go:117] "RemoveContainer" containerID="41e7d35dfdcae674041139ee119cbb4d317e4e2a359ad7c7b25f2f0204cfe1ec"
Sep 16 18:18:42.599797 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.599780    2554 scope.go:117] "RemoveContainer" containerID="41e7d35dfdcae674041139ee119cbb4d317e4e2a359ad7c7b25f2f0204cfe1ec"
Sep 16 18:18:42.600103 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:18:42.600081    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e7d35dfdcae674041139ee119cbb4d317e4e2a359ad7c7b25f2f0204cfe1ec\": container with ID starting with 41e7d35dfdcae674041139ee119cbb4d317e4e2a359ad7c7b25f2f0204cfe1ec not found: ID does not exist" containerID="41e7d35dfdcae674041139ee119cbb4d317e4e2a359ad7c7b25f2f0204cfe1ec"
Sep 16 18:18:42.600171 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.600115    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e7d35dfdcae674041139ee119cbb4d317e4e2a359ad7c7b25f2f0204cfe1ec"} err="failed to get container status \"41e7d35dfdcae674041139ee119cbb4d317e4e2a359ad7c7b25f2f0204cfe1ec\": rpc error: code = NotFound desc = could not find container \"41e7d35dfdcae674041139ee119cbb4d317e4e2a359ad7c7b25f2f0204cfe1ec\": container with ID starting with 41e7d35dfdcae674041139ee119cbb4d317e4e2a359ad7c7b25f2f0204cfe1ec not found: ID does not exist"
Sep 16 18:18:42.611604 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.611581    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cfb89dbc-l9d7f"]
Sep 16 18:18:42.615418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.615397    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6cfb89dbc-l9d7f"]
Sep 16 18:18:42.882335 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:18:42.882257    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9812ded9-6393-45db-9c21-338afa68c6b9" path="/var/lib/kubelet/pods/9812ded9-6393-45db-9c21-338afa68c6b9/volumes"
Sep 16 18:19:09.147104 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:09.147024    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:09.165202 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:09.165175    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:09.708212 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:09.708184    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:22.656878 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:22.656835    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:19:22.657321 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:22.657099    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="alertmanager" containerID="cri-o://7078849921a9dfad0f6e4879a25af35169955bf3f287d387c1d3c822788ef3bb" gracePeriod=120
Sep 16 18:19:22.657321 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:22.657125    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="kube-rbac-proxy-web" containerID="cri-o://c587e0e10955c4737dd0c3ff6a7ea7049e4b82c6d609c3f104d996d08a119e71" gracePeriod=120
Sep 16 18:19:22.657321 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:22.657146    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="prom-label-proxy" containerID="cri-o://a95bd5b7242d6867b7272c0933e35d8fdc0519e3e5c5ede44f1631b7acc9e974" gracePeriod=120
Sep 16 18:19:22.657321 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:22.657158    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="kube-rbac-proxy-metric" containerID="cri-o://0987ea6e18c2200161c2d8d12c0674402e30432244a58da9f7f07bb37fe68120" gracePeriod=120
Sep 16 18:19:22.657321 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:22.657110    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="kube-rbac-proxy" containerID="cri-o://7500f371705e62af9f0ba972936a24d885081ba01502c9ddebc9255dd71efe36" gracePeriod=120
Sep 16 18:19:22.657521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:22.657180    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="config-reloader" containerID="cri-o://2af5cdb3cc8e64b91166a74031e6837cb19194d0755ffb5ade43a85cb72041da" gracePeriod=120
Sep 16 18:19:23.751167 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.751131    2554 generic.go:356] "Generic (PLEG): container finished" podID="f02331d3-c795-4f3b-9949-20b462c55065" containerID="a95bd5b7242d6867b7272c0933e35d8fdc0519e3e5c5ede44f1631b7acc9e974" exitCode=0
Sep 16 18:19:23.751167 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.751157    2554 generic.go:356] "Generic (PLEG): container finished" podID="f02331d3-c795-4f3b-9949-20b462c55065" containerID="0987ea6e18c2200161c2d8d12c0674402e30432244a58da9f7f07bb37fe68120" exitCode=0
Sep 16 18:19:23.751167 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.751164    2554 generic.go:356] "Generic (PLEG): container finished" podID="f02331d3-c795-4f3b-9949-20b462c55065" containerID="7500f371705e62af9f0ba972936a24d885081ba01502c9ddebc9255dd71efe36" exitCode=0
Sep 16 18:19:23.751167 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.751170    2554 generic.go:356] "Generic (PLEG): container finished" podID="f02331d3-c795-4f3b-9949-20b462c55065" containerID="2af5cdb3cc8e64b91166a74031e6837cb19194d0755ffb5ade43a85cb72041da" exitCode=0
Sep 16 18:19:23.751564 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.751175    2554 generic.go:356] "Generic (PLEG): container finished" podID="f02331d3-c795-4f3b-9949-20b462c55065" containerID="7078849921a9dfad0f6e4879a25af35169955bf3f287d387c1d3c822788ef3bb" exitCode=0
Sep 16 18:19:23.751564 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.751203    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f02331d3-c795-4f3b-9949-20b462c55065","Type":"ContainerDied","Data":"a95bd5b7242d6867b7272c0933e35d8fdc0519e3e5c5ede44f1631b7acc9e974"}
Sep 16 18:19:23.751564 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.751289    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f02331d3-c795-4f3b-9949-20b462c55065","Type":"ContainerDied","Data":"0987ea6e18c2200161c2d8d12c0674402e30432244a58da9f7f07bb37fe68120"}
Sep 16 18:19:23.751564 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.751316    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f02331d3-c795-4f3b-9949-20b462c55065","Type":"ContainerDied","Data":"7500f371705e62af9f0ba972936a24d885081ba01502c9ddebc9255dd71efe36"}
Sep 16 18:19:23.751564 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.751339    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f02331d3-c795-4f3b-9949-20b462c55065","Type":"ContainerDied","Data":"2af5cdb3cc8e64b91166a74031e6837cb19194d0755ffb5ade43a85cb72041da"}
Sep 16 18:19:23.751564 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.751363    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f02331d3-c795-4f3b-9949-20b462c55065","Type":"ContainerDied","Data":"7078849921a9dfad0f6e4879a25af35169955bf3f287d387c1d3c822788ef3bb"}
Sep 16 18:19:23.887122 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.887096    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:23.936361 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.936328    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-config-volume\") pod \"f02331d3-c795-4f3b-9949-20b462c55065\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") "
Sep 16 18:19:23.936514 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.936387    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f02331d3-c795-4f3b-9949-20b462c55065-alertmanager-main-db\") pod \"f02331d3-c795-4f3b-9949-20b462c55065\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") "
Sep 16 18:19:23.936514 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.936418    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpn76\" (UniqueName: \"kubernetes.io/projected/f02331d3-c795-4f3b-9949-20b462c55065-kube-api-access-fpn76\") pod \"f02331d3-c795-4f3b-9949-20b462c55065\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") "
Sep 16 18:19:23.936514 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.936444    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-main-tls\") pod \"f02331d3-c795-4f3b-9949-20b462c55065\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") "
Sep 16 18:19:23.936658 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.936511    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f02331d3-c795-4f3b-9949-20b462c55065-alertmanager-trusted-ca-bundle\") pod \"f02331d3-c795-4f3b-9949-20b462c55065\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") "
Sep 16 18:19:23.936658 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.936536    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-web-config\") pod \"f02331d3-c795-4f3b-9949-20b462c55065\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") "
Sep 16 18:19:23.936658 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.936578    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-cluster-tls-config\") pod \"f02331d3-c795-4f3b-9949-20b462c55065\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") "
Sep 16 18:19:23.936658 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.936602    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f02331d3-c795-4f3b-9949-20b462c55065-metrics-client-ca\") pod \"f02331d3-c795-4f3b-9949-20b462c55065\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") "
Sep 16 18:19:23.936658 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.936631    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f02331d3-c795-4f3b-9949-20b462c55065-config-out\") pod \"f02331d3-c795-4f3b-9949-20b462c55065\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") "
Sep 16 18:19:23.936658 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.936655    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f02331d3-c795-4f3b-9949-20b462c55065\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") "
Sep 16 18:19:23.936954 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.936689    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy-web\") pod \"f02331d3-c795-4f3b-9949-20b462c55065\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") "
Sep 16 18:19:23.936954 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.936712    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f02331d3-c795-4f3b-9949-20b462c55065-tls-assets\") pod \"f02331d3-c795-4f3b-9949-20b462c55065\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") "
Sep 16 18:19:23.936954 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.936777    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy\") pod \"f02331d3-c795-4f3b-9949-20b462c55065\" (UID: \"f02331d3-c795-4f3b-9949-20b462c55065\") "
Sep 16 18:19:23.938036 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.936773    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02331d3-c795-4f3b-9949-20b462c55065-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f02331d3-c795-4f3b-9949-20b462c55065" (UID: "f02331d3-c795-4f3b-9949-20b462c55065"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:19:23.938036 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.937039    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02331d3-c795-4f3b-9949-20b462c55065-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f02331d3-c795-4f3b-9949-20b462c55065" (UID: "f02331d3-c795-4f3b-9949-20b462c55065"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:19:23.938036 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.937426    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02331d3-c795-4f3b-9949-20b462c55065-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f02331d3-c795-4f3b-9949-20b462c55065" (UID: "f02331d3-c795-4f3b-9949-20b462c55065"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:19:23.940163 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.939488    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f02331d3-c795-4f3b-9949-20b462c55065" (UID: "f02331d3-c795-4f3b-9949-20b462c55065"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:19:23.940163 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.939828    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-config-volume" (OuterVolumeSpecName: "config-volume") pod "f02331d3-c795-4f3b-9949-20b462c55065" (UID: "f02331d3-c795-4f3b-9949-20b462c55065"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:19:23.940163 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.939855    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f02331d3-c795-4f3b-9949-20b462c55065" (UID: "f02331d3-c795-4f3b-9949-20b462c55065"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:19:23.941624 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.941585    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02331d3-c795-4f3b-9949-20b462c55065-kube-api-access-fpn76" (OuterVolumeSpecName: "kube-api-access-fpn76") pod "f02331d3-c795-4f3b-9949-20b462c55065" (UID: "f02331d3-c795-4f3b-9949-20b462c55065"). InnerVolumeSpecName "kube-api-access-fpn76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:19:23.941941 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.941841    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02331d3-c795-4f3b-9949-20b462c55065-config-out" (OuterVolumeSpecName: "config-out") pod "f02331d3-c795-4f3b-9949-20b462c55065" (UID: "f02331d3-c795-4f3b-9949-20b462c55065"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:19:23.941941 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.941912    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f02331d3-c795-4f3b-9949-20b462c55065" (UID: "f02331d3-c795-4f3b-9949-20b462c55065"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:19:23.943529 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.943493    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02331d3-c795-4f3b-9949-20b462c55065-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f02331d3-c795-4f3b-9949-20b462c55065" (UID: "f02331d3-c795-4f3b-9949-20b462c55065"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:19:23.943822 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.943796    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f02331d3-c795-4f3b-9949-20b462c55065" (UID: "f02331d3-c795-4f3b-9949-20b462c55065"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:19:23.947587 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.947560    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f02331d3-c795-4f3b-9949-20b462c55065" (UID: "f02331d3-c795-4f3b-9949-20b462c55065"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:19:23.955225 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:23.955096    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-web-config" (OuterVolumeSpecName: "web-config") pod "f02331d3-c795-4f3b-9949-20b462c55065" (UID: "f02331d3-c795-4f3b-9949-20b462c55065"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:19:24.038343 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.038297    2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:24.038343 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.038333    2554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f02331d3-c795-4f3b-9949-20b462c55065-tls-assets\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:24.038343 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.038343    2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:24.038343 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.038352    2554 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:24.038622 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.038363    2554 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f02331d3-c795-4f3b-9949-20b462c55065-alertmanager-main-db\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:24.038622 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.038372    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpn76\" (UniqueName: \"kubernetes.io/projected/f02331d3-c795-4f3b-9949-20b462c55065-kube-api-access-fpn76\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:24.038622 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.038382    2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-main-tls\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:24.038622 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.038390    2554 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f02331d3-c795-4f3b-9949-20b462c55065-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:24.038622 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.038399    2554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-web-config\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:24.038622 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.038407    2554 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-cluster-tls-config\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:24.038622 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.038414    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f02331d3-c795-4f3b-9949-20b462c55065-metrics-client-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:24.038622 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.038422    2554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f02331d3-c795-4f3b-9949-20b462c55065-config-out\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:24.038622 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.038430    2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f02331d3-c795-4f3b-9949-20b462c55065-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:24.758781 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.758747    2554 generic.go:356] "Generic (PLEG): container finished" podID="f02331d3-c795-4f3b-9949-20b462c55065" containerID="c587e0e10955c4737dd0c3ff6a7ea7049e4b82c6d609c3f104d996d08a119e71" exitCode=0
Sep 16 18:19:24.759128 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.758810    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f02331d3-c795-4f3b-9949-20b462c55065","Type":"ContainerDied","Data":"c587e0e10955c4737dd0c3ff6a7ea7049e4b82c6d609c3f104d996d08a119e71"}
Sep 16 18:19:24.759128 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.758817    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:24.759128 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.758877    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f02331d3-c795-4f3b-9949-20b462c55065","Type":"ContainerDied","Data":"9747b9ad76294f3e87d85f719994a01f111072aad30045992ec277e3f494e533"}
Sep 16 18:19:24.759128 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.758912    2554 scope.go:117] "RemoveContainer" containerID="a95bd5b7242d6867b7272c0933e35d8fdc0519e3e5c5ede44f1631b7acc9e974"
Sep 16 18:19:24.766559 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.766540    2554 scope.go:117] "RemoveContainer" containerID="0987ea6e18c2200161c2d8d12c0674402e30432244a58da9f7f07bb37fe68120"
Sep 16 18:19:24.773706 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.773535    2554 scope.go:117] "RemoveContainer" containerID="7500f371705e62af9f0ba972936a24d885081ba01502c9ddebc9255dd71efe36"
Sep 16 18:19:24.780024 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.780008    2554 scope.go:117] "RemoveContainer" containerID="c587e0e10955c4737dd0c3ff6a7ea7049e4b82c6d609c3f104d996d08a119e71"
Sep 16 18:19:24.785101 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.785076    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:19:24.788108 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.788085    2554 scope.go:117] "RemoveContainer" containerID="2af5cdb3cc8e64b91166a74031e6837cb19194d0755ffb5ade43a85cb72041da"
Sep 16 18:19:24.789098 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.789081    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:19:24.794821 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.794803    2554 scope.go:117] "RemoveContainer" containerID="7078849921a9dfad0f6e4879a25af35169955bf3f287d387c1d3c822788ef3bb"
Sep 16 18:19:24.801325 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.801306    2554 scope.go:117] "RemoveContainer" containerID="9384ccfe408a7e5ee2642207045a1aab25c932341437fc4117a8376f9f75e7be"
Sep 16 18:19:24.807469 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.807454    2554 scope.go:117] "RemoveContainer" containerID="a95bd5b7242d6867b7272c0933e35d8fdc0519e3e5c5ede44f1631b7acc9e974"
Sep 16 18:19:24.807730 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.807711    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95bd5b7242d6867b7272c0933e35d8fdc0519e3e5c5ede44f1631b7acc9e974\": container with ID starting with a95bd5b7242d6867b7272c0933e35d8fdc0519e3e5c5ede44f1631b7acc9e974 not found: ID does not exist" containerID="a95bd5b7242d6867b7272c0933e35d8fdc0519e3e5c5ede44f1631b7acc9e974"
Sep 16 18:19:24.807773 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.807738    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95bd5b7242d6867b7272c0933e35d8fdc0519e3e5c5ede44f1631b7acc9e974"} err="failed to get container status \"a95bd5b7242d6867b7272c0933e35d8fdc0519e3e5c5ede44f1631b7acc9e974\": rpc error: code = NotFound desc = could not find container \"a95bd5b7242d6867b7272c0933e35d8fdc0519e3e5c5ede44f1631b7acc9e974\": container with ID starting with a95bd5b7242d6867b7272c0933e35d8fdc0519e3e5c5ede44f1631b7acc9e974 not found: ID does not exist"
Sep 16 18:19:24.807773 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.807757    2554 scope.go:117] "RemoveContainer" containerID="0987ea6e18c2200161c2d8d12c0674402e30432244a58da9f7f07bb37fe68120"
Sep 16 18:19:24.807986 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.807969    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0987ea6e18c2200161c2d8d12c0674402e30432244a58da9f7f07bb37fe68120\": container with ID starting with 0987ea6e18c2200161c2d8d12c0674402e30432244a58da9f7f07bb37fe68120 not found: ID does not exist" containerID="0987ea6e18c2200161c2d8d12c0674402e30432244a58da9f7f07bb37fe68120"
Sep 16 18:19:24.808054 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.807996    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0987ea6e18c2200161c2d8d12c0674402e30432244a58da9f7f07bb37fe68120"} err="failed to get container status \"0987ea6e18c2200161c2d8d12c0674402e30432244a58da9f7f07bb37fe68120\": rpc error: code = NotFound desc = could not find container \"0987ea6e18c2200161c2d8d12c0674402e30432244a58da9f7f07bb37fe68120\": container with ID starting with 0987ea6e18c2200161c2d8d12c0674402e30432244a58da9f7f07bb37fe68120 not found: ID does not exist"
Sep 16 18:19:24.808054 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.808020    2554 scope.go:117] "RemoveContainer" containerID="7500f371705e62af9f0ba972936a24d885081ba01502c9ddebc9255dd71efe36"
Sep 16 18:19:24.808269 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.808244    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7500f371705e62af9f0ba972936a24d885081ba01502c9ddebc9255dd71efe36\": container with ID starting with 7500f371705e62af9f0ba972936a24d885081ba01502c9ddebc9255dd71efe36 not found: ID does not exist" containerID="7500f371705e62af9f0ba972936a24d885081ba01502c9ddebc9255dd71efe36"
Sep 16 18:19:24.808359 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.808273    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7500f371705e62af9f0ba972936a24d885081ba01502c9ddebc9255dd71efe36"} err="failed to get container status \"7500f371705e62af9f0ba972936a24d885081ba01502c9ddebc9255dd71efe36\": rpc error: code = NotFound desc = could not find container \"7500f371705e62af9f0ba972936a24d885081ba01502c9ddebc9255dd71efe36\": container with ID starting with 7500f371705e62af9f0ba972936a24d885081ba01502c9ddebc9255dd71efe36 not found: ID does not exist"
Sep 16 18:19:24.808359 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.808294    2554 scope.go:117] "RemoveContainer" containerID="c587e0e10955c4737dd0c3ff6a7ea7049e4b82c6d609c3f104d996d08a119e71"
Sep 16 18:19:24.808546 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.808528    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c587e0e10955c4737dd0c3ff6a7ea7049e4b82c6d609c3f104d996d08a119e71\": container with ID starting with c587e0e10955c4737dd0c3ff6a7ea7049e4b82c6d609c3f104d996d08a119e71 not found: ID does not exist" containerID="c587e0e10955c4737dd0c3ff6a7ea7049e4b82c6d609c3f104d996d08a119e71"
Sep 16 18:19:24.808632 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.808550    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c587e0e10955c4737dd0c3ff6a7ea7049e4b82c6d609c3f104d996d08a119e71"} err="failed to get container status \"c587e0e10955c4737dd0c3ff6a7ea7049e4b82c6d609c3f104d996d08a119e71\": rpc error: code = NotFound desc = could not find container \"c587e0e10955c4737dd0c3ff6a7ea7049e4b82c6d609c3f104d996d08a119e71\": container with ID starting with c587e0e10955c4737dd0c3ff6a7ea7049e4b82c6d609c3f104d996d08a119e71 not found: ID does not exist"
Sep 16 18:19:24.808632 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.808563    2554 scope.go:117] "RemoveContainer" containerID="2af5cdb3cc8e64b91166a74031e6837cb19194d0755ffb5ade43a85cb72041da"
Sep 16 18:19:24.808813 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.808796    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af5cdb3cc8e64b91166a74031e6837cb19194d0755ffb5ade43a85cb72041da\": container with ID starting with 2af5cdb3cc8e64b91166a74031e6837cb19194d0755ffb5ade43a85cb72041da not found: ID does not exist" containerID="2af5cdb3cc8e64b91166a74031e6837cb19194d0755ffb5ade43a85cb72041da"
Sep 16 18:19:24.808855 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.808816    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af5cdb3cc8e64b91166a74031e6837cb19194d0755ffb5ade43a85cb72041da"} err="failed to get container status \"2af5cdb3cc8e64b91166a74031e6837cb19194d0755ffb5ade43a85cb72041da\": rpc error: code = NotFound desc = could not find container \"2af5cdb3cc8e64b91166a74031e6837cb19194d0755ffb5ade43a85cb72041da\": container with ID starting with 2af5cdb3cc8e64b91166a74031e6837cb19194d0755ffb5ade43a85cb72041da not found: ID does not exist"
Sep 16 18:19:24.808855 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.808833    2554 scope.go:117] "RemoveContainer" containerID="7078849921a9dfad0f6e4879a25af35169955bf3f287d387c1d3c822788ef3bb"
Sep 16 18:19:24.809050 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.809035    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7078849921a9dfad0f6e4879a25af35169955bf3f287d387c1d3c822788ef3bb\": container with ID starting with 7078849921a9dfad0f6e4879a25af35169955bf3f287d387c1d3c822788ef3bb not found: ID does not exist" containerID="7078849921a9dfad0f6e4879a25af35169955bf3f287d387c1d3c822788ef3bb"
Sep 16 18:19:24.809088 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.809056    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7078849921a9dfad0f6e4879a25af35169955bf3f287d387c1d3c822788ef3bb"} err="failed to get container status \"7078849921a9dfad0f6e4879a25af35169955bf3f287d387c1d3c822788ef3bb\": rpc error: code = NotFound desc = could not find container \"7078849921a9dfad0f6e4879a25af35169955bf3f287d387c1d3c822788ef3bb\": container with ID starting with 7078849921a9dfad0f6e4879a25af35169955bf3f287d387c1d3c822788ef3bb not found: ID does not exist"
Sep 16 18:19:24.809088 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.809069    2554 scope.go:117] "RemoveContainer" containerID="9384ccfe408a7e5ee2642207045a1aab25c932341437fc4117a8376f9f75e7be"
Sep 16 18:19:24.809271 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.809257    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9384ccfe408a7e5ee2642207045a1aab25c932341437fc4117a8376f9f75e7be\": container with ID starting with 9384ccfe408a7e5ee2642207045a1aab25c932341437fc4117a8376f9f75e7be not found: ID does not exist" containerID="9384ccfe408a7e5ee2642207045a1aab25c932341437fc4117a8376f9f75e7be"
Sep 16 18:19:24.809307 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.809274    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9384ccfe408a7e5ee2642207045a1aab25c932341437fc4117a8376f9f75e7be"} err="failed to get container status \"9384ccfe408a7e5ee2642207045a1aab25c932341437fc4117a8376f9f75e7be\": rpc error: code = NotFound desc = could not find container \"9384ccfe408a7e5ee2642207045a1aab25c932341437fc4117a8376f9f75e7be\": container with ID starting with 9384ccfe408a7e5ee2642207045a1aab25c932341437fc4117a8376f9f75e7be not found: ID does not exist"
Sep 16 18:19:24.822705 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.822684    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:19:24.822949 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.822935    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9812ded9-6393-45db-9c21-338afa68c6b9" containerName="registry"
Sep 16 18:19:24.822998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.822951    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9812ded9-6393-45db-9c21-338afa68c6b9" containerName="registry"
Sep 16 18:19:24.822998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.822960    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="kube-rbac-proxy"
Sep 16 18:19:24.822998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.822967    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="kube-rbac-proxy"
Sep 16 18:19:24.822998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.822982    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="config-reloader"
Sep 16 18:19:24.822998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.822987    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="config-reloader"
Sep 16 18:19:24.822998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.822995    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="init-config-reloader"
Sep 16 18:19:24.822998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.823000    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="init-config-reloader"
Sep 16 18:19:24.823198 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.823011    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="kube-rbac-proxy-metric"
Sep 16 18:19:24.823198 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.823019    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="kube-rbac-proxy-metric"
Sep 16 18:19:24.823198 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.823025    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="kube-rbac-proxy-web"
Sep 16 18:19:24.823198 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.823030    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="kube-rbac-proxy-web"
Sep 16 18:19:24.823198 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.823038    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="alertmanager"
Sep 16 18:19:24.823198 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.823042    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="alertmanager"
Sep 16 18:19:24.823198 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.823048    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="prom-label-proxy"
Sep 16 18:19:24.823198 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.823053    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="prom-label-proxy"
Sep 16 18:19:24.823198 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.823096    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="alertmanager"
Sep 16 18:19:24.823198 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.823104    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="config-reloader"
Sep 16 18:19:24.823198 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.823111    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9812ded9-6393-45db-9c21-338afa68c6b9" containerName="registry"
Sep 16 18:19:24.823198 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.823116    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="kube-rbac-proxy-web"
Sep 16 18:19:24.823198 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.823120    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="prom-label-proxy"
Sep 16 18:19:24.823198 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.823125    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="kube-rbac-proxy"
Sep 16 18:19:24.823198 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.823130    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f02331d3-c795-4f3b-9949-20b462c55065" containerName="kube-rbac-proxy-metric"
Sep 16 18:19:24.826747 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.826732    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:24.830442 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.830420    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-web-config\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" type="*v1.Secret"
Sep 16 18:19:24.830582 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.830560    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" type="*v1.Secret"
Sep 16 18:19:24.830582 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.830567    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-cluster-tls-config\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" type="*v1.Secret"
Sep 16 18:19:24.830697 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.830662    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"alertmanager-main-0\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" podUID="403782cd-2a03-492e-a394-f483a9db911d" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:24.830804 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.830775    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-tls-assets-0\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" type="*v1.Secret"
Sep 16 18:19:24.831061 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.831038    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-tls\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" type="*v1.Secret"
Sep 16 18:19:24.831195 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.831171    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-dockercfg-sn2np\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sn2np\"" type="*v1.Secret"
Sep 16 18:19:24.831195 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.831177    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-kube-rbac-proxy-metric\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" type="*v1.Secret"
Sep 16 18:19:24.831309 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.831253    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-main-generated\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" type="*v1.Secret"
Sep 16 18:19:24.831309 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.831262    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"alertmanager-trusted-ca-bundle\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" type="*v1.ConfigMap"
Sep 16 18:19:24.831381 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:24.831306    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"alertmanager-kube-rbac-proxy-web\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" type="*v1.Secret"
Sep 16 18:19:24.847167 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.847083    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:19:24.882527 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.882495    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f02331d3-c795-4f3b-9949-20b462c55065" path="/var/lib/kubelet/pods/f02331d3-c795-4f3b-9949-20b462c55065/volumes"
Sep 16 18:19:24.946341 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.946310    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-web-config\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:24.946487 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.946348    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-config-volume\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:24.946487 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.946371    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/403782cd-2a03-492e-a394-f483a9db911d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:24.946487 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.946432    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bll8t\" (UniqueName: \"kubernetes.io/projected/403782cd-2a03-492e-a394-f483a9db911d-kube-api-access-bll8t\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:24.946487 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.946456    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:24.946610 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.946511    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:24.946610 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.946535    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/403782cd-2a03-492e-a394-f483a9db911d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:24.946610 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.946570    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/403782cd-2a03-492e-a394-f483a9db911d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:24.946610 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.946601    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:24.946719 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.946617    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/403782cd-2a03-492e-a394-f483a9db911d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:24.946719 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.946633    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:24.946719 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.946653    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:24.946719 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:24.946709    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/403782cd-2a03-492e-a394-f483a9db911d-config-out\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.047436 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.047340    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bll8t\" (UniqueName: \"kubernetes.io/projected/403782cd-2a03-492e-a394-f483a9db911d-kube-api-access-bll8t\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.047436 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.047396    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.047436 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.047418    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.047436 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.047433    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/403782cd-2a03-492e-a394-f483a9db911d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.047708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.047451    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/403782cd-2a03-492e-a394-f483a9db911d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.047708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.047466    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.047708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.047481    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/403782cd-2a03-492e-a394-f483a9db911d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.047708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.047583    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.047708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.047633    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.047708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.047671    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/403782cd-2a03-492e-a394-f483a9db911d-config-out\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.047976 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.047718    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-web-config\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.047976 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.047752    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-config-volume\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.047976 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.047780    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/403782cd-2a03-492e-a394-f483a9db911d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.048352 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.048332    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/403782cd-2a03-492e-a394-f483a9db911d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.048496 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.048471    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/403782cd-2a03-492e-a394-f483a9db911d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.049784 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.049767    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/403782cd-2a03-492e-a394-f483a9db911d-config-out\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.055763 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.055744    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bll8t\" (UniqueName: \"kubernetes.io/projected/403782cd-2a03-492e-a394-f483a9db911d-kube-api-access-bll8t\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.736291 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.736256    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:19:25.740607 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.740583    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-web-config\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.763210 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.763182    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sn2np\""
Sep 16 18:19:25.805761 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.805718    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:19:25.808408 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.808379    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/403782cd-2a03-492e-a394-f483a9db911d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.831843 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.831809    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:19:25.840289 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.840264    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/403782cd-2a03-492e-a394-f483a9db911d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.858411 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.858382    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:19:25.870067 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.870043    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:25.886198 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.886174    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:19:25.890297 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:25.890270    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:26.048023 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:26.047916    2554 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-generated: failed to sync secret cache: timed out waiting for the condition
Sep 16 18:19:26.048023 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:26.047960    2554 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-cluster-tls-config: failed to sync secret cache: timed out waiting for the condition
Sep 16 18:19:26.048023 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:26.047924    2554 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-kube-rbac-proxy: failed to sync secret cache: timed out waiting for the condition
Sep 16 18:19:26.048023 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:26.048011    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-config-volume podName:403782cd-2a03-492e-a394-f483a9db911d nodeName:}" failed. No retries permitted until 2026-09-16 18:19:26.547991402 +0000 UTC m=+190.107159217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "config-volume" (UniqueName: "kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-config-volume") pod "alertmanager-main-0" (UID: "403782cd-2a03-492e-a394-f483a9db911d") : failed to sync secret cache: timed out waiting for the condition
Sep 16 18:19:26.048023 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:26.048034    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-cluster-tls-config podName:403782cd-2a03-492e-a394-f483a9db911d nodeName:}" failed. No retries permitted until 2026-09-16 18:19:26.548025843 +0000 UTC m=+190.107193654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-tls-config" (UniqueName: "kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-cluster-tls-config") pod "alertmanager-main-0" (UID: "403782cd-2a03-492e-a394-f483a9db911d") : failed to sync secret cache: timed out waiting for the condition
Sep 16 18:19:26.048415 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:26.048057    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-kube-rbac-proxy podName:403782cd-2a03-492e-a394-f483a9db911d nodeName:}" failed. No retries permitted until 2026-09-16 18:19:26.5480439 +0000 UTC m=+190.107211716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-kube-rbac-proxy" (UniqueName: "kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-kube-rbac-proxy") pod "alertmanager-main-0" (UID: "403782cd-2a03-492e-a394-f483a9db911d") : failed to sync secret cache: timed out waiting for the condition
Sep 16 18:19:26.048415 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:26.047933    2554 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: failed to sync secret cache: timed out waiting for the condition
Sep 16 18:19:26.048415 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:26.048086    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-main-tls podName:403782cd-2a03-492e-a394-f483a9db911d nodeName:}" failed. No retries permitted until 2026-09-16 18:19:26.548080272 +0000 UTC m=+190.107248124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "403782cd-2a03-492e-a394-f483a9db911d") : failed to sync secret cache: timed out waiting for the condition
Sep 16 18:19:26.057701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:26.057679    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:19:26.089757 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:26.089730    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:19:26.347126 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:26.347036    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:19:26.431361 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:26.431327    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:19:26.559215 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:26.559181    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:26.559215 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:26.559218    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:26.559406 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:26.559276    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-config-volume\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:26.559406 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:26.559321    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:26.561745 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:26.561715    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:26.561873 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:26.561821    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:26.561873 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:26.561830    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:26.561873 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:26.561829    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/403782cd-2a03-492e-a394-f483a9db911d-config-volume\") pod \"alertmanager-main-0\" (UID: \"403782cd-2a03-492e-a394-f483a9db911d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:26.635904 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:26.635804    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:19:26.973797 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:26.973771    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:19:26.975989 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:19:26.975960    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403782cd_2a03_492e_a394_f483a9db911d.slice/crio-c802fd5a6095e5db6d22601f9b697aeb551ba1364dbca33257193ba96366dc9f WatchSource:0}: Error finding container c802fd5a6095e5db6d22601f9b697aeb551ba1364dbca33257193ba96366dc9f: Status 404 returned error can't find the container with id c802fd5a6095e5db6d22601f9b697aeb551ba1364dbca33257193ba96366dc9f
Sep 16 18:19:27.010611 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.010579    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:19:27.010879 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.010845    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="prometheus" containerID="cri-o://c656937d4db909565db02fa8dcc7e618e18d6ae9ebe0fa2c26589baaf10a7a2c" gracePeriod=600
Sep 16 18:19:27.010991 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.010903    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="thanos-sidecar" containerID="cri-o://afe7ad10cd143b4fd4ee2eb00c26fec7e6e0ed55bfb2c45e3342a847b287e2a8" gracePeriod=600
Sep 16 18:19:27.010991 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.010933    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="config-reloader" containerID="cri-o://e88247dc0b010bf07302d88598d555c1115d64b3e5cc458a3d1a1864cd811c6d" gracePeriod=600
Sep 16 18:19:27.010991 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.010938    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="kube-rbac-proxy-web" containerID="cri-o://2b955ec62ac652801e15d150c79b3064531be65efe77dc0e62d909b657215305" gracePeriod=600
Sep 16 18:19:27.011166 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.010910    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://30dadc025eb64419180e1f17723378f692715dab4df203f05d5adff2c34a28c8" gracePeriod=600
Sep 16 18:19:27.011166 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.011060    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="kube-rbac-proxy" containerID="cri-o://3979ad91ecccdd2e49c70bff4ee9dec99fe0b0fcd05541dae39efa95efcbf944" gracePeriod=600
Sep 16 18:19:27.771740 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.771703    2554 generic.go:356] "Generic (PLEG): container finished" podID="403782cd-2a03-492e-a394-f483a9db911d" containerID="2a70df4eed8b47143207f80a8607670807de17e0f557880e8b4cc1396a1eecee" exitCode=0
Sep 16 18:19:27.771886 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.771789    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"403782cd-2a03-492e-a394-f483a9db911d","Type":"ContainerDied","Data":"2a70df4eed8b47143207f80a8607670807de17e0f557880e8b4cc1396a1eecee"}
Sep 16 18:19:27.771886 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.771864    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"403782cd-2a03-492e-a394-f483a9db911d","Type":"ContainerStarted","Data":"c802fd5a6095e5db6d22601f9b697aeb551ba1364dbca33257193ba96366dc9f"}
Sep 16 18:19:27.775377 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.775353    2554 generic.go:356] "Generic (PLEG): container finished" podID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerID="30dadc025eb64419180e1f17723378f692715dab4df203f05d5adff2c34a28c8" exitCode=0
Sep 16 18:19:27.775377 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.775372    2554 generic.go:356] "Generic (PLEG): container finished" podID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerID="3979ad91ecccdd2e49c70bff4ee9dec99fe0b0fcd05541dae39efa95efcbf944" exitCode=0
Sep 16 18:19:27.775377 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.775378    2554 generic.go:356] "Generic (PLEG): container finished" podID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerID="afe7ad10cd143b4fd4ee2eb00c26fec7e6e0ed55bfb2c45e3342a847b287e2a8" exitCode=0
Sep 16 18:19:27.775573 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.775385    2554 generic.go:356] "Generic (PLEG): container finished" podID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerID="e88247dc0b010bf07302d88598d555c1115d64b3e5cc458a3d1a1864cd811c6d" exitCode=0
Sep 16 18:19:27.775573 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.775391    2554 generic.go:356] "Generic (PLEG): container finished" podID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerID="c656937d4db909565db02fa8dcc7e618e18d6ae9ebe0fa2c26589baaf10a7a2c" exitCode=0
Sep 16 18:19:27.775573 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.775417    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddeab720-40e2-43e3-be8c-c6457f48905d","Type":"ContainerDied","Data":"30dadc025eb64419180e1f17723378f692715dab4df203f05d5adff2c34a28c8"}
Sep 16 18:19:27.775573 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.775463    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddeab720-40e2-43e3-be8c-c6457f48905d","Type":"ContainerDied","Data":"3979ad91ecccdd2e49c70bff4ee9dec99fe0b0fcd05541dae39efa95efcbf944"}
Sep 16 18:19:27.775573 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.775500    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddeab720-40e2-43e3-be8c-c6457f48905d","Type":"ContainerDied","Data":"afe7ad10cd143b4fd4ee2eb00c26fec7e6e0ed55bfb2c45e3342a847b287e2a8"}
Sep 16 18:19:27.775573 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.775525    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddeab720-40e2-43e3-be8c-c6457f48905d","Type":"ContainerDied","Data":"e88247dc0b010bf07302d88598d555c1115d64b3e5cc458a3d1a1864cd811c6d"}
Sep 16 18:19:27.775573 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:27.775549    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddeab720-40e2-43e3-be8c-c6457f48905d","Type":"ContainerDied","Data":"c656937d4db909565db02fa8dcc7e618e18d6ae9ebe0fa2c26589baaf10a7a2c"}
Sep 16 18:19:28.243881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.243856    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.273943 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.273873    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-grpc-tls\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.273943 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.273919    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9dst\" (UniqueName: \"kubernetes.io/projected/ddeab720-40e2-43e3-be8c-c6457f48905d-kube-api-access-x9dst\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274147 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.273947    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-config\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274147 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.273984    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-kubelet-serving-ca-bundle\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274147 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274016    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-tls\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274147 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274040    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274147 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274070    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-1\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274147 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274099    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddeab720-40e2-43e3-be8c-c6457f48905d-tls-assets\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274147 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274125    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-serving-certs-ca-bundle\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274147 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274149    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-thanos-prometheus-http-client-file\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274177    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-kube-rbac-proxy\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274224    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-metrics-client-certs\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274272    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-0\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274301    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274337    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-db\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274366    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-trusted-ca-bundle\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274399    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddeab720-40e2-43e3-be8c-c6457f48905d-config-out\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274434    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-metrics-client-ca\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274464    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-web-config\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274496    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-2\") pod \"ddeab720-40e2-43e3-be8c-c6457f48905d\" (UID: \"ddeab720-40e2-43e3-be8c-c6457f48905d\") "
Sep 16 18:19:28.274550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274512    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:19:28.275060 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274678    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:19:28.275060 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.274824    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:19:28.275569 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.275541    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:19:28.275909 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.275887    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:19:28.276552 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.276305    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:19:28.276552 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.276417    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:19:28.277379 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.277360    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.277567 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.277554    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.277855 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.277839    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.277977 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.277805    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:19:28.278084 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.277887    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-config" (OuterVolumeSpecName: "config") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:19:28.278275 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.277936    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:19:28.278383 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.278221    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:19:28.278473 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.278353    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddeab720-40e2-43e3-be8c-c6457f48905d-config-out" (OuterVolumeSpecName: "config-out") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:19:28.278566 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.278446    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:19:28.278718 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.278696    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:19:28.278802 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.278780    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:19:28.278878 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.278854    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:19:28.278916 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.278861    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddeab720-40e2-43e3-be8c-c6457f48905d-kube-api-access-x9dst" (OuterVolumeSpecName: "kube-api-access-x9dst") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "kube-api-access-x9dst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:19:28.279460 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.279447    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:19:28.279512 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.279500    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddeab720-40e2-43e3-be8c-c6457f48905d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:19:28.287936 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.287907    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-web-config" (OuterVolumeSpecName: "web-config") pod "ddeab720-40e2-43e3-be8c-c6457f48905d" (UID: "ddeab720-40e2-43e3-be8c-c6457f48905d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:19:28.378601 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378558    2554 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-metrics-client-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.378601 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378605    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.378792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378621    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.378792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378637    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-k8s-db\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.378792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378653    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.378792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378669    2554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddeab720-40e2-43e3-be8c-c6457f48905d-config-out\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.378792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378681    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-metrics-client-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.378792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378694    2554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-web-config\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.378792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378706    2554 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-grpc-tls\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.378792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378717    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9dst\" (UniqueName: \"kubernetes.io/projected/ddeab720-40e2-43e3-be8c-c6457f48905d-kube-api-access-x9dst\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.378792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378729    2554 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-config\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.378792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378741    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.378792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378755    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.378792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378768    2554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddeab720-40e2-43e3-be8c-c6457f48905d-tls-assets\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.378792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378781    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddeab720-40e2-43e3-be8c-c6457f48905d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.378792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378795    2554 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.379211 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.378808    2554 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddeab720-40e2-43e3-be8c-c6457f48905d-secret-kube-rbac-proxy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:19:28.782984 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.782939    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"403782cd-2a03-492e-a394-f483a9db911d","Type":"ContainerStarted","Data":"7379dc171090fb7cc5bc109816dcfda6c5cbc304cff36955f0436f98cdd720c8"}
Sep 16 18:19:28.783150 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.783026    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"403782cd-2a03-492e-a394-f483a9db911d","Type":"ContainerStarted","Data":"e2de550f4759181c2438b34e0d8bfeda14cbff8aa6165fb9cb07cb2517289055"}
Sep 16 18:19:28.783150 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.783064    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"403782cd-2a03-492e-a394-f483a9db911d","Type":"ContainerStarted","Data":"7f431c5f8d1420ad22871baf2de4d7be3d2dd57ea2b786405cc95e703f69472d"}
Sep 16 18:19:28.783150 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.783099    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"403782cd-2a03-492e-a394-f483a9db911d","Type":"ContainerStarted","Data":"100c2cec5978e2dc6d9a694ffc2290a0f3b446540c4b732bf132e092b2841004"}
Sep 16 18:19:28.783150 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.783133    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"403782cd-2a03-492e-a394-f483a9db911d","Type":"ContainerStarted","Data":"2eb417afc243cd5dc091dae2f937b4c7163a6d6dcd5de2b74a53b7ce4666d27f"}
Sep 16 18:19:28.783384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.783169    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"403782cd-2a03-492e-a394-f483a9db911d","Type":"ContainerStarted","Data":"8cb3ea6d265a32bde89974d3b832346e7f51ed50fbeb90146915ec40ff4ee360"}
Sep 16 18:19:28.786139 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.786113    2554 generic.go:356] "Generic (PLEG): container finished" podID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerID="2b955ec62ac652801e15d150c79b3064531be65efe77dc0e62d909b657215305" exitCode=0
Sep 16 18:19:28.786256 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.786178    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.786256 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.786177    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddeab720-40e2-43e3-be8c-c6457f48905d","Type":"ContainerDied","Data":"2b955ec62ac652801e15d150c79b3064531be65efe77dc0e62d909b657215305"}
Sep 16 18:19:28.786331 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.786293    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddeab720-40e2-43e3-be8c-c6457f48905d","Type":"ContainerDied","Data":"e3de0265243e0b8674c42e4927ae8996d7b49eba6082d3ed27f1c8e4b72955bb"}
Sep 16 18:19:28.786331 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.786325    2554 scope.go:117] "RemoveContainer" containerID="30dadc025eb64419180e1f17723378f692715dab4df203f05d5adff2c34a28c8"
Sep 16 18:19:28.794878 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.794857    2554 scope.go:117] "RemoveContainer" containerID="3979ad91ecccdd2e49c70bff4ee9dec99fe0b0fcd05541dae39efa95efcbf944"
Sep 16 18:19:28.802102 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.802082    2554 scope.go:117] "RemoveContainer" containerID="2b955ec62ac652801e15d150c79b3064531be65efe77dc0e62d909b657215305"
Sep 16 18:19:28.809000 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.808981    2554 scope.go:117] "RemoveContainer" containerID="afe7ad10cd143b4fd4ee2eb00c26fec7e6e0ed55bfb2c45e3342a847b287e2a8"
Sep 16 18:19:28.815903 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.815879    2554 scope.go:117] "RemoveContainer" containerID="e88247dc0b010bf07302d88598d555c1115d64b3e5cc458a3d1a1864cd811c6d"
Sep 16 18:19:28.822677 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.822650    2554 scope.go:117] "RemoveContainer" containerID="c656937d4db909565db02fa8dcc7e618e18d6ae9ebe0fa2c26589baaf10a7a2c"
Sep 16 18:19:28.829862 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.829844    2554 scope.go:117] "RemoveContainer" containerID="bdb3cf80bc15aefd6f288b83955552055edd867427882721f807400f600cace6"
Sep 16 18:19:28.837470 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.837442    2554 scope.go:117] "RemoveContainer" containerID="30dadc025eb64419180e1f17723378f692715dab4df203f05d5adff2c34a28c8"
Sep 16 18:19:28.837817 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:28.837798    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30dadc025eb64419180e1f17723378f692715dab4df203f05d5adff2c34a28c8\": container with ID starting with 30dadc025eb64419180e1f17723378f692715dab4df203f05d5adff2c34a28c8 not found: ID does not exist" containerID="30dadc025eb64419180e1f17723378f692715dab4df203f05d5adff2c34a28c8"
Sep 16 18:19:28.837884 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.837826    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30dadc025eb64419180e1f17723378f692715dab4df203f05d5adff2c34a28c8"} err="failed to get container status \"30dadc025eb64419180e1f17723378f692715dab4df203f05d5adff2c34a28c8\": rpc error: code = NotFound desc = could not find container \"30dadc025eb64419180e1f17723378f692715dab4df203f05d5adff2c34a28c8\": container with ID starting with 30dadc025eb64419180e1f17723378f692715dab4df203f05d5adff2c34a28c8 not found: ID does not exist"
Sep 16 18:19:28.837884 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.837848    2554 scope.go:117] "RemoveContainer" containerID="3979ad91ecccdd2e49c70bff4ee9dec99fe0b0fcd05541dae39efa95efcbf944"
Sep 16 18:19:28.838137 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:28.838119    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3979ad91ecccdd2e49c70bff4ee9dec99fe0b0fcd05541dae39efa95efcbf944\": container with ID starting with 3979ad91ecccdd2e49c70bff4ee9dec99fe0b0fcd05541dae39efa95efcbf944 not found: ID does not exist" containerID="3979ad91ecccdd2e49c70bff4ee9dec99fe0b0fcd05541dae39efa95efcbf944"
Sep 16 18:19:28.838178 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.838141    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3979ad91ecccdd2e49c70bff4ee9dec99fe0b0fcd05541dae39efa95efcbf944"} err="failed to get container status \"3979ad91ecccdd2e49c70bff4ee9dec99fe0b0fcd05541dae39efa95efcbf944\": rpc error: code = NotFound desc = could not find container \"3979ad91ecccdd2e49c70bff4ee9dec99fe0b0fcd05541dae39efa95efcbf944\": container with ID starting with 3979ad91ecccdd2e49c70bff4ee9dec99fe0b0fcd05541dae39efa95efcbf944 not found: ID does not exist"
Sep 16 18:19:28.838178 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.838160    2554 scope.go:117] "RemoveContainer" containerID="2b955ec62ac652801e15d150c79b3064531be65efe77dc0e62d909b657215305"
Sep 16 18:19:28.838454 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:28.838436    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b955ec62ac652801e15d150c79b3064531be65efe77dc0e62d909b657215305\": container with ID starting with 2b955ec62ac652801e15d150c79b3064531be65efe77dc0e62d909b657215305 not found: ID does not exist" containerID="2b955ec62ac652801e15d150c79b3064531be65efe77dc0e62d909b657215305"
Sep 16 18:19:28.838514 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.838459    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b955ec62ac652801e15d150c79b3064531be65efe77dc0e62d909b657215305"} err="failed to get container status \"2b955ec62ac652801e15d150c79b3064531be65efe77dc0e62d909b657215305\": rpc error: code = NotFound desc = could not find container \"2b955ec62ac652801e15d150c79b3064531be65efe77dc0e62d909b657215305\": container with ID starting with 2b955ec62ac652801e15d150c79b3064531be65efe77dc0e62d909b657215305 not found: ID does not exist"
Sep 16 18:19:28.838514 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.838471    2554 scope.go:117] "RemoveContainer" containerID="afe7ad10cd143b4fd4ee2eb00c26fec7e6e0ed55bfb2c45e3342a847b287e2a8"
Sep 16 18:19:28.838731 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:28.838711    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe7ad10cd143b4fd4ee2eb00c26fec7e6e0ed55bfb2c45e3342a847b287e2a8\": container with ID starting with afe7ad10cd143b4fd4ee2eb00c26fec7e6e0ed55bfb2c45e3342a847b287e2a8 not found: ID does not exist" containerID="afe7ad10cd143b4fd4ee2eb00c26fec7e6e0ed55bfb2c45e3342a847b287e2a8"
Sep 16 18:19:28.838772 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.838734    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe7ad10cd143b4fd4ee2eb00c26fec7e6e0ed55bfb2c45e3342a847b287e2a8"} err="failed to get container status \"afe7ad10cd143b4fd4ee2eb00c26fec7e6e0ed55bfb2c45e3342a847b287e2a8\": rpc error: code = NotFound desc = could not find container \"afe7ad10cd143b4fd4ee2eb00c26fec7e6e0ed55bfb2c45e3342a847b287e2a8\": container with ID starting with afe7ad10cd143b4fd4ee2eb00c26fec7e6e0ed55bfb2c45e3342a847b287e2a8 not found: ID does not exist"
Sep 16 18:19:28.838772 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.838746    2554 scope.go:117] "RemoveContainer" containerID="e88247dc0b010bf07302d88598d555c1115d64b3e5cc458a3d1a1864cd811c6d"
Sep 16 18:19:28.838882 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.838838    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.822537727 podStartE2EDuration="4.838824782s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.287055ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:19:28.835718984 +0000 UTC m=+192.394886849" watchObservedRunningTime="2026-09-16 18:19:28.838824782 +0000 UTC m=+192.397992615"
Sep 16 18:19:28.838990 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:28.838977    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88247dc0b010bf07302d88598d555c1115d64b3e5cc458a3d1a1864cd811c6d\": container with ID starting with e88247dc0b010bf07302d88598d555c1115d64b3e5cc458a3d1a1864cd811c6d not found: ID does not exist" containerID="e88247dc0b010bf07302d88598d555c1115d64b3e5cc458a3d1a1864cd811c6d"
Sep 16 18:19:28.839025 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.838996    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88247dc0b010bf07302d88598d555c1115d64b3e5cc458a3d1a1864cd811c6d"} err="failed to get container status \"e88247dc0b010bf07302d88598d555c1115d64b3e5cc458a3d1a1864cd811c6d\": rpc error: code = NotFound desc = could not find container \"e88247dc0b010bf07302d88598d555c1115d64b3e5cc458a3d1a1864cd811c6d\": container with ID starting with e88247dc0b010bf07302d88598d555c1115d64b3e5cc458a3d1a1864cd811c6d not found: ID does not exist"
Sep 16 18:19:28.839025 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.839009    2554 scope.go:117] "RemoveContainer" containerID="c656937d4db909565db02fa8dcc7e618e18d6ae9ebe0fa2c26589baaf10a7a2c"
Sep 16 18:19:28.839296 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:28.839206    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c656937d4db909565db02fa8dcc7e618e18d6ae9ebe0fa2c26589baaf10a7a2c\": container with ID starting with c656937d4db909565db02fa8dcc7e618e18d6ae9ebe0fa2c26589baaf10a7a2c not found: ID does not exist" containerID="c656937d4db909565db02fa8dcc7e618e18d6ae9ebe0fa2c26589baaf10a7a2c"
Sep 16 18:19:28.839296 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.839257    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c656937d4db909565db02fa8dcc7e618e18d6ae9ebe0fa2c26589baaf10a7a2c"} err="failed to get container status \"c656937d4db909565db02fa8dcc7e618e18d6ae9ebe0fa2c26589baaf10a7a2c\": rpc error: code = NotFound desc = could not find container \"c656937d4db909565db02fa8dcc7e618e18d6ae9ebe0fa2c26589baaf10a7a2c\": container with ID starting with c656937d4db909565db02fa8dcc7e618e18d6ae9ebe0fa2c26589baaf10a7a2c not found: ID does not exist"
Sep 16 18:19:28.839296 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.839278    2554 scope.go:117] "RemoveContainer" containerID="bdb3cf80bc15aefd6f288b83955552055edd867427882721f807400f600cace6"
Sep 16 18:19:28.839558 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:19:28.839540    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb3cf80bc15aefd6f288b83955552055edd867427882721f807400f600cace6\": container with ID starting with bdb3cf80bc15aefd6f288b83955552055edd867427882721f807400f600cace6 not found: ID does not exist" containerID="bdb3cf80bc15aefd6f288b83955552055edd867427882721f807400f600cace6"
Sep 16 18:19:28.839591 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.839563    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb3cf80bc15aefd6f288b83955552055edd867427882721f807400f600cace6"} err="failed to get container status \"bdb3cf80bc15aefd6f288b83955552055edd867427882721f807400f600cace6\": rpc error: code = NotFound desc = could not find container \"bdb3cf80bc15aefd6f288b83955552055edd867427882721f807400f600cace6\": container with ID starting with bdb3cf80bc15aefd6f288b83955552055edd867427882721f807400f600cace6 not found: ID does not exist"
Sep 16 18:19:28.866597 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.861601    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:19:28.870437 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.870404    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:19:28.883298 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.883270    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" path="/var/lib/kubelet/pods/ddeab720-40e2-43e3-be8c-c6457f48905d/volumes"
Sep 16 18:19:28.898015 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.897985    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:19:28.898280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898263    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="prometheus"
Sep 16 18:19:28.898329 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898283    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="prometheus"
Sep 16 18:19:28.898329 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898293    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="config-reloader"
Sep 16 18:19:28.898329 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898299    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="config-reloader"
Sep 16 18:19:28.898329 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898310    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="init-config-reloader"
Sep 16 18:19:28.898329 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898315    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="init-config-reloader"
Sep 16 18:19:28.898329 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898326    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="thanos-sidecar"
Sep 16 18:19:28.898329 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898331    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="thanos-sidecar"
Sep 16 18:19:28.898521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898337    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="kube-rbac-proxy-web"
Sep 16 18:19:28.898521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898342    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="kube-rbac-proxy-web"
Sep 16 18:19:28.898521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898350    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="kube-rbac-proxy"
Sep 16 18:19:28.898521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898355    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="kube-rbac-proxy"
Sep 16 18:19:28.898521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898361    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="kube-rbac-proxy-thanos"
Sep 16 18:19:28.898521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898367    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="kube-rbac-proxy-thanos"
Sep 16 18:19:28.898521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898416    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="kube-rbac-proxy-thanos"
Sep 16 18:19:28.898521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898423    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="thanos-sidecar"
Sep 16 18:19:28.898521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898429    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="kube-rbac-proxy"
Sep 16 18:19:28.898521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898435    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="kube-rbac-proxy-web"
Sep 16 18:19:28.898521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898441    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="config-reloader"
Sep 16 18:19:28.898521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.898448    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddeab720-40e2-43e3-be8c-c6457f48905d" containerName="prometheus"
Sep 16 18:19:28.901818 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.901799    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.905562 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.905291    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:19:28.905562 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.905308    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:19:28.905562 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.905292    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:19:28.905562 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.905377    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:19:28.905562 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.905400    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:19:28.905562 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.905466    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:19:28.905889 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.905660    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:19:28.906163 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.906145    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:19:28.906316 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.906292    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:19:28.906316 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.906292    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bws42\""
Sep 16 18:19:28.906479 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.906370    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:19:28.906757 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.906740    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bmpohrcrmq569\""
Sep 16 18:19:28.906936 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.906919    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:19:28.906936 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.906927    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:19:28.907056 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.906927    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:19:28.908515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.908495    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:19:28.913901 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.913879    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:19:28.920804 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.920779    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:19:28.982160 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982124    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982160 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982162    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54zw6\" (UniqueName: \"kubernetes.io/projected/7b7f4318-80cc-40ac-b06e-5250fa2a9730-kube-api-access-54zw6\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982410 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982182    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982410 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982200    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b7f4318-80cc-40ac-b06e-5250fa2a9730-config-out\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982410 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982217    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-web-config\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982410 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982301    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b7f4318-80cc-40ac-b06e-5250fa2a9730-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982410 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982357    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982410 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982390    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982410 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982405    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982423    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982450    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982466    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7b7f4318-80cc-40ac-b06e-5250fa2a9730-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982538    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982579    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982624    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982650    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982675    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982963 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982712    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982963 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982753    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-config\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:28.982963 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:28.982779    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.083580 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083480    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.083580 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083528    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.083580 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083548    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.083845 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083588    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.083845 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083609    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.083845 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083640    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.083845 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083665    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-config\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.083845 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083693    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.083845 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083722    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.083845 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083747    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54zw6\" (UniqueName: \"kubernetes.io/projected/7b7f4318-80cc-40ac-b06e-5250fa2a9730-kube-api-access-54zw6\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.083845 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083774    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.083845 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083804    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b7f4318-80cc-40ac-b06e-5250fa2a9730-config-out\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.083845 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083828    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-web-config\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.084402 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083854    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b7f4318-80cc-40ac-b06e-5250fa2a9730-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.084402 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083893    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.084402 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083925    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.084402 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083948    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.084402 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.083977    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.084402 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.084014    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.084402 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.084037    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7b7f4318-80cc-40ac-b06e-5250fa2a9730-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.084402 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.084358    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7b7f4318-80cc-40ac-b06e-5250fa2a9730-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.084750 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.084550    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.085190 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.085160    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.085832 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.085455    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.086979 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.086661    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.086979 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.086673    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.086979 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.086937    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.087691 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.087663    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.087793 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.087690    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7b7f4318-80cc-40ac-b06e-5250fa2a9730-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.087911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.087865    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-config\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.087911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.087887    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.088386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.088183    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b7f4318-80cc-40ac-b06e-5250fa2a9730-config-out\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.088998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.088608    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.088998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.088721    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-web-config\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.088998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.088731    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.088998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.088902    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b7f4318-80cc-40ac-b06e-5250fa2a9730-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.089545 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.089523    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.089986 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.089966    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.090034 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.090024    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7b7f4318-80cc-40ac-b06e-5250fa2a9730-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.093417 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.093396    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54zw6\" (UniqueName: \"kubernetes.io/projected/7b7f4318-80cc-40ac-b06e-5250fa2a9730-kube-api-access-54zw6\") pod \"prometheus-k8s-0\" (UID: \"7b7f4318-80cc-40ac-b06e-5250fa2a9730\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.215918 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.215869    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:19:29.501126 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.501096    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:19:29.502769 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:19:29.502738    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7f4318_80cc_40ac_b06e_5250fa2a9730.slice/crio-5d1dbcf92b7fdf9ff26438d6245a814c1336cc4bd34e974b515c236d235b4397 WatchSource:0}: Error finding container 5d1dbcf92b7fdf9ff26438d6245a814c1336cc4bd34e974b515c236d235b4397: Status 404 returned error can't find the container with id 5d1dbcf92b7fdf9ff26438d6245a814c1336cc4bd34e974b515c236d235b4397
Sep 16 18:19:29.793365 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.793275    2554 generic.go:356] "Generic (PLEG): container finished" podID="7b7f4318-80cc-40ac-b06e-5250fa2a9730" containerID="e5d0b5bc19d90e6cb3aa62f0f237757b14f270fe7fb994f3a3f4b7cfc0a025e5" exitCode=0
Sep 16 18:19:29.793492 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.793368    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b7f4318-80cc-40ac-b06e-5250fa2a9730","Type":"ContainerDied","Data":"e5d0b5bc19d90e6cb3aa62f0f237757b14f270fe7fb994f3a3f4b7cfc0a025e5"}
Sep 16 18:19:29.793492 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:29.793435    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b7f4318-80cc-40ac-b06e-5250fa2a9730","Type":"ContainerStarted","Data":"5d1dbcf92b7fdf9ff26438d6245a814c1336cc4bd34e974b515c236d235b4397"}
Sep 16 18:19:30.802760 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:30.802724    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b7f4318-80cc-40ac-b06e-5250fa2a9730","Type":"ContainerStarted","Data":"922c097ebc7ca966b8526c75a04480014da2acb59e6e81e69de73b673673f26b"}
Sep 16 18:19:30.803128 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:30.802811    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b7f4318-80cc-40ac-b06e-5250fa2a9730","Type":"ContainerStarted","Data":"a2930b4da0a925e7f126b33bd7835005dd8c9cd63f0d464a52a6e0432e478c39"}
Sep 16 18:19:30.803128 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:30.802836    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b7f4318-80cc-40ac-b06e-5250fa2a9730","Type":"ContainerStarted","Data":"b6a0076b9e5baf9d7b9e25c8783349edf8904ccc6ed99d816d330b1a431fe751"}
Sep 16 18:19:30.803128 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:30.802863    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b7f4318-80cc-40ac-b06e-5250fa2a9730","Type":"ContainerStarted","Data":"774a21c25dbbf31478d9a88cba934048c6dfae109c7990ed4307b4dbbaf13f43"}
Sep 16 18:19:30.803128 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:30.802884    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b7f4318-80cc-40ac-b06e-5250fa2a9730","Type":"ContainerStarted","Data":"d449b2b096fb73c536d569a167e2db1a4d63f6260a2959d09f4e36f6b9e0ee24"}
Sep 16 18:19:30.803128 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:30.802906    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7b7f4318-80cc-40ac-b06e-5250fa2a9730","Type":"ContainerStarted","Data":"2efcd463a001981a8d55e91df143b0ab9d279ef2600574766dc8311200ad9947"}
Sep 16 18:19:30.840050 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:30.839966    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.81816243 podStartE2EDuration="2.839948364s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.785934ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:19:30.838410938 +0000 UTC m=+194.397578771" watchObservedRunningTime="2026-09-16 18:19:30.839948364 +0000 UTC m=+194.399116198"
Sep 16 18:19:34.216386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:19:34.216347    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:20:29.216284 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:20:29.216218    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:20:29.231148 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:20:29.231123    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:20:30.036277 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:20:30.036250    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:21:16.759272 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:21:16.759241    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:21:16.763586 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:21:16.763555    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:21:16.769184 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:21:16.769157    2554 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:22:40.975873 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:40.975799    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m4558"]
Sep 16 18:22:40.978901 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:40.978883    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m4558"
Sep 16 18:22:40.981570 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:40.981551    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:22:40.982720 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:40.982693    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 18:22:40.982823 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:40.982728    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-nf75k\""
Sep 16 18:22:40.988950 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:40.988931    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m4558"]
Sep 16 18:22:41.173781 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:41.173747    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbqsx\" (UniqueName: \"kubernetes.io/projected/0d001f5b-8956-4d13-9be3-ba7abe8c65ff-kube-api-access-pbqsx\") pod \"openshift-lws-operator-fd8ccff4c-m4558\" (UID: \"0d001f5b-8956-4d13-9be3-ba7abe8c65ff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m4558"
Sep 16 18:22:41.173781 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:41.173785    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d001f5b-8956-4d13-9be3-ba7abe8c65ff-tmp\") pod \"openshift-lws-operator-fd8ccff4c-m4558\" (UID: \"0d001f5b-8956-4d13-9be3-ba7abe8c65ff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m4558"
Sep 16 18:22:41.274351 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:41.274320    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbqsx\" (UniqueName: \"kubernetes.io/projected/0d001f5b-8956-4d13-9be3-ba7abe8c65ff-kube-api-access-pbqsx\") pod \"openshift-lws-operator-fd8ccff4c-m4558\" (UID: \"0d001f5b-8956-4d13-9be3-ba7abe8c65ff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m4558"
Sep 16 18:22:41.274498 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:41.274357    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d001f5b-8956-4d13-9be3-ba7abe8c65ff-tmp\") pod \"openshift-lws-operator-fd8ccff4c-m4558\" (UID: \"0d001f5b-8956-4d13-9be3-ba7abe8c65ff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m4558"
Sep 16 18:22:41.274706 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:41.274691    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d001f5b-8956-4d13-9be3-ba7abe8c65ff-tmp\") pod \"openshift-lws-operator-fd8ccff4c-m4558\" (UID: \"0d001f5b-8956-4d13-9be3-ba7abe8c65ff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m4558"
Sep 16 18:22:41.286348 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:41.286325    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbqsx\" (UniqueName: \"kubernetes.io/projected/0d001f5b-8956-4d13-9be3-ba7abe8c65ff-kube-api-access-pbqsx\") pod \"openshift-lws-operator-fd8ccff4c-m4558\" (UID: \"0d001f5b-8956-4d13-9be3-ba7abe8c65ff\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m4558"
Sep 16 18:22:41.297067 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:41.297048    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m4558"
Sep 16 18:22:41.575425 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:41.575351    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m4558"]
Sep 16 18:22:41.581156 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:22:41.581129    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d001f5b_8956_4d13_9be3_ba7abe8c65ff.slice/crio-ad8fd6f8df885627d18f0bc10115d17541149399e8709cad7db055c484108bfb WatchSource:0}: Error finding container ad8fd6f8df885627d18f0bc10115d17541149399e8709cad7db055c484108bfb: Status 404 returned error can't find the container with id ad8fd6f8df885627d18f0bc10115d17541149399e8709cad7db055c484108bfb
Sep 16 18:22:41.582331 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:41.582312    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:22:42.505319 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:42.505284    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m4558" event={"ID":"0d001f5b-8956-4d13-9be3-ba7abe8c65ff","Type":"ContainerStarted","Data":"ad8fd6f8df885627d18f0bc10115d17541149399e8709cad7db055c484108bfb"}
Sep 16 18:22:44.516247 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:44.516201    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m4558" event={"ID":"0d001f5b-8956-4d13-9be3-ba7abe8c65ff","Type":"ContainerStarted","Data":"22f1c3bfa3d78977edabf63b0cf06d3dd63ee8964dbdbd6b4e0c73e6c68001f7"}
Sep 16 18:22:44.537328 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:22:44.537275    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m4558" podStartSLOduration=1.941393704 podStartE2EDuration="4.537261086s" totalImagesPullingTime="2.595867382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:22:44.535583692 +0000 UTC m=+388.094751525" watchObservedRunningTime="2026-09-16 18:22:44.537261086 +0000 UTC m=+388.096428920"
Sep 16 18:23:04.853656 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:04.853619    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"]
Sep 16 18:23:04.858063 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:04.858046    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:04.861437 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:04.861415    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 18:23:04.861649 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:04.861631    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 18:23:04.861722 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:04.861672    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 18:23:04.862017 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:04.862000    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5ghg2\""
Sep 16 18:23:04.862603 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:04.862586    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 18:23:04.877324 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:04.877306    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"]
Sep 16 18:23:04.966515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:04.966476    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c77db88a-0d6c-4094-ad8e-f25cbefc7bc5-tmp\") pod \"opendatahub-operator-controller-manager-86bdbbf975-w4fdb\" (UID: \"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5\") " pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:04.966704 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:04.966526    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c77db88a-0d6c-4094-ad8e-f25cbefc7bc5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86bdbbf975-w4fdb\" (UID: \"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5\") " pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:04.966704 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:04.966570    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c77db88a-0d6c-4094-ad8e-f25cbefc7bc5-webhook-cert\") pod \"opendatahub-operator-controller-manager-86bdbbf975-w4fdb\" (UID: \"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5\") " pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:04.966704 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:04.966593    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8rw2\" (UniqueName: \"kubernetes.io/projected/c77db88a-0d6c-4094-ad8e-f25cbefc7bc5-kube-api-access-x8rw2\") pod \"opendatahub-operator-controller-manager-86bdbbf975-w4fdb\" (UID: \"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5\") " pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:04.966704 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:04.966620    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c77db88a-0d6c-4094-ad8e-f25cbefc7bc5-manifests\") pod \"opendatahub-operator-controller-manager-86bdbbf975-w4fdb\" (UID: \"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5\") " pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:05.067899 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:05.067867    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c77db88a-0d6c-4094-ad8e-f25cbefc7bc5-tmp\") pod \"opendatahub-operator-controller-manager-86bdbbf975-w4fdb\" (UID: \"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5\") " pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:05.068063 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:05.067906    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c77db88a-0d6c-4094-ad8e-f25cbefc7bc5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86bdbbf975-w4fdb\" (UID: \"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5\") " pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:05.068063 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:05.067925    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c77db88a-0d6c-4094-ad8e-f25cbefc7bc5-webhook-cert\") pod \"opendatahub-operator-controller-manager-86bdbbf975-w4fdb\" (UID: \"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5\") " pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:05.068063 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:05.067941    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8rw2\" (UniqueName: \"kubernetes.io/projected/c77db88a-0d6c-4094-ad8e-f25cbefc7bc5-kube-api-access-x8rw2\") pod \"opendatahub-operator-controller-manager-86bdbbf975-w4fdb\" (UID: \"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5\") " pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:05.068063 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:05.067964    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c77db88a-0d6c-4094-ad8e-f25cbefc7bc5-manifests\") pod \"opendatahub-operator-controller-manager-86bdbbf975-w4fdb\" (UID: \"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5\") " pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:05.068460 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:05.068433    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c77db88a-0d6c-4094-ad8e-f25cbefc7bc5-manifests\") pod \"opendatahub-operator-controller-manager-86bdbbf975-w4fdb\" (UID: \"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5\") " pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:05.070107 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:05.070080    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c77db88a-0d6c-4094-ad8e-f25cbefc7bc5-tmp\") pod \"opendatahub-operator-controller-manager-86bdbbf975-w4fdb\" (UID: \"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5\") " pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:05.070369 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:05.070349    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c77db88a-0d6c-4094-ad8e-f25cbefc7bc5-webhook-cert\") pod \"opendatahub-operator-controller-manager-86bdbbf975-w4fdb\" (UID: \"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5\") " pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:05.070443 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:05.070354    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c77db88a-0d6c-4094-ad8e-f25cbefc7bc5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86bdbbf975-w4fdb\" (UID: \"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5\") " pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:05.080379 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:05.080357    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8rw2\" (UniqueName: \"kubernetes.io/projected/c77db88a-0d6c-4094-ad8e-f25cbefc7bc5-kube-api-access-x8rw2\") pod \"opendatahub-operator-controller-manager-86bdbbf975-w4fdb\" (UID: \"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5\") " pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:05.167726 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:05.167640    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:05.453056 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:05.452980    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"]
Sep 16 18:23:05.456578 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:23:05.456554    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77db88a_0d6c_4094_ad8e_f25cbefc7bc5.slice/crio-9807228a52fb95600227681ddefb9eeba812bff242572f99a3f8a7fd78e051d5 WatchSource:0}: Error finding container 9807228a52fb95600227681ddefb9eeba812bff242572f99a3f8a7fd78e051d5: Status 404 returned error can't find the container with id 9807228a52fb95600227681ddefb9eeba812bff242572f99a3f8a7fd78e051d5
Sep 16 18:23:05.593664 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:05.593626    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb" event={"ID":"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5","Type":"ContainerStarted","Data":"9807228a52fb95600227681ddefb9eeba812bff242572f99a3f8a7fd78e051d5"}
Sep 16 18:23:08.607781 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:08.607687    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb" event={"ID":"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5","Type":"ContainerStarted","Data":"8c4bf05352998e8d1bcb737de0b3469544ad9c3c5f66024c44a1d37fa895df2f"}
Sep 16 18:23:09.613639 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:09.613598    2554 generic.go:356] "Generic (PLEG): container finished" podID="c77db88a-0d6c-4094-ad8e-f25cbefc7bc5" containerID="8c4bf05352998e8d1bcb737de0b3469544ad9c3c5f66024c44a1d37fa895df2f" exitCode=0
Sep 16 18:23:09.614003 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:09.613655    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb" event={"ID":"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5","Type":"ContainerDied","Data":"8c4bf05352998e8d1bcb737de0b3469544ad9c3c5f66024c44a1d37fa895df2f"}
Sep 16 18:23:14.635680 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:14.635644    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb" event={"ID":"c77db88a-0d6c-4094-ad8e-f25cbefc7bc5","Type":"ContainerStarted","Data":"05f0b9b17580e2076de310432c07d3e8d9032d699cc5516c80d52e3379aff6f6"}
Sep 16 18:23:14.636089 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:14.635782    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:14.673661 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:14.673619    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb" podStartSLOduration=2.252030072 podStartE2EDuration="10.673604015s" totalImagesPullingTime="7.34837791s" totalInitContainerRuntime="1.073196033s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:23:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:23:14.671391535 +0000 UTC m=+418.230559370" watchObservedRunningTime="2026-09-16 18:23:14.673604015 +0000 UTC m=+418.232771848"
Sep 16 18:23:25.642162 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:25.642126    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-86bdbbf975-w4fdb"
Sep 16 18:23:38.183971 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.183935    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mmq55"]
Sep 16 18:23:38.189155 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.189130    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mmq55"
Sep 16 18:23:38.191423 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.191399    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 16 18:23:38.191529 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.191468    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-gbntb\""
Sep 16 18:23:38.197973 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.197947    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mmq55"]
Sep 16 18:23:38.242053 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.242019    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz8dr\" (UniqueName: \"kubernetes.io/projected/5c40361f-f58e-4143-80d6-c2a70e17acdb-kube-api-access-lz8dr\") pod \"odh-model-controller-7998c67dc4-mmq55\" (UID: \"5c40361f-f58e-4143-80d6-c2a70e17acdb\") " pod="opendatahub/odh-model-controller-7998c67dc4-mmq55"
Sep 16 18:23:38.242259 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.242067    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c40361f-f58e-4143-80d6-c2a70e17acdb-cert\") pod \"odh-model-controller-7998c67dc4-mmq55\" (UID: \"5c40361f-f58e-4143-80d6-c2a70e17acdb\") " pod="opendatahub/odh-model-controller-7998c67dc4-mmq55"
Sep 16 18:23:38.342894 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.342846    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz8dr\" (UniqueName: \"kubernetes.io/projected/5c40361f-f58e-4143-80d6-c2a70e17acdb-kube-api-access-lz8dr\") pod \"odh-model-controller-7998c67dc4-mmq55\" (UID: \"5c40361f-f58e-4143-80d6-c2a70e17acdb\") " pod="opendatahub/odh-model-controller-7998c67dc4-mmq55"
Sep 16 18:23:38.342894 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.342905    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c40361f-f58e-4143-80d6-c2a70e17acdb-cert\") pod \"odh-model-controller-7998c67dc4-mmq55\" (UID: \"5c40361f-f58e-4143-80d6-c2a70e17acdb\") " pod="opendatahub/odh-model-controller-7998c67dc4-mmq55"
Sep 16 18:23:38.343142 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:23:38.343036    2554 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 16 18:23:38.343142 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:23:38.343086    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c40361f-f58e-4143-80d6-c2a70e17acdb-cert podName:5c40361f-f58e-4143-80d6-c2a70e17acdb nodeName:}" failed. No retries permitted until 2026-09-16 18:23:38.843071803 +0000 UTC m=+442.402239628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5c40361f-f58e-4143-80d6-c2a70e17acdb-cert") pod "odh-model-controller-7998c67dc4-mmq55" (UID: "5c40361f-f58e-4143-80d6-c2a70e17acdb") : secret "odh-model-controller-webhook-cert" not found
Sep 16 18:23:38.351015 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.350993    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz8dr\" (UniqueName: \"kubernetes.io/projected/5c40361f-f58e-4143-80d6-c2a70e17acdb-kube-api-access-lz8dr\") pod \"odh-model-controller-7998c67dc4-mmq55\" (UID: \"5c40361f-f58e-4143-80d6-c2a70e17acdb\") " pod="opendatahub/odh-model-controller-7998c67dc4-mmq55"
Sep 16 18:23:38.846385 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.846353    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c40361f-f58e-4143-80d6-c2a70e17acdb-cert\") pod \"odh-model-controller-7998c67dc4-mmq55\" (UID: \"5c40361f-f58e-4143-80d6-c2a70e17acdb\") " pod="opendatahub/odh-model-controller-7998c67dc4-mmq55"
Sep 16 18:23:38.848824 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.848800    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c40361f-f58e-4143-80d6-c2a70e17acdb-cert\") pod \"odh-model-controller-7998c67dc4-mmq55\" (UID: \"5c40361f-f58e-4143-80d6-c2a70e17acdb\") " pod="opendatahub/odh-model-controller-7998c67dc4-mmq55"
Sep 16 18:23:38.883033 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.883004    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rxrjz"]
Sep 16 18:23:38.886207 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.886192    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-rxrjz"
Sep 16 18:23:38.888693 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.888676    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 16 18:23:38.888760 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.888717    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-hdsxl\""
Sep 16 18:23:38.892567 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.892540    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rxrjz"]
Sep 16 18:23:38.946672 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.946635    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kfzw\" (UniqueName: \"kubernetes.io/projected/d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978-kube-api-access-2kfzw\") pod \"model-serving-api-54d89bd5fc-rxrjz\" (UID: \"d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978\") " pod="opendatahub/model-serving-api-54d89bd5fc-rxrjz"
Sep 16 18:23:38.946844 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:38.946701    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978-tls-certs\") pod \"model-serving-api-54d89bd5fc-rxrjz\" (UID: \"d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978\") " pod="opendatahub/model-serving-api-54d89bd5fc-rxrjz"
Sep 16 18:23:39.047754 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:39.047719    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kfzw\" (UniqueName: \"kubernetes.io/projected/d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978-kube-api-access-2kfzw\") pod \"model-serving-api-54d89bd5fc-rxrjz\" (UID: \"d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978\") " pod="opendatahub/model-serving-api-54d89bd5fc-rxrjz"
Sep 16 18:23:39.047931 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:39.047790    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978-tls-certs\") pod \"model-serving-api-54d89bd5fc-rxrjz\" (UID: \"d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978\") " pod="opendatahub/model-serving-api-54d89bd5fc-rxrjz"
Sep 16 18:23:39.047931 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:23:39.047905    2554 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 16 18:23:39.048005 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:23:39.047967    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978-tls-certs podName:d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978 nodeName:}" failed. No retries permitted until 2026-09-16 18:23:39.547948764 +0000 UTC m=+443.107116577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978-tls-certs") pod "model-serving-api-54d89bd5fc-rxrjz" (UID: "d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978") : secret "model-serving-api-tls" not found
Sep 16 18:23:39.056262 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:39.056211    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kfzw\" (UniqueName: \"kubernetes.io/projected/d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978-kube-api-access-2kfzw\") pod \"model-serving-api-54d89bd5fc-rxrjz\" (UID: \"d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978\") " pod="opendatahub/model-serving-api-54d89bd5fc-rxrjz"
Sep 16 18:23:39.099930 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:39.099834    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mmq55"
Sep 16 18:23:39.371678 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:39.371603    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mmq55"]
Sep 16 18:23:39.374620 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:23:39.374591    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c40361f_f58e_4143_80d6_c2a70e17acdb.slice/crio-09c719d628bc52fbfaaa0f0aea422c95821d690f5e43923b49daf7d1b2107b9c WatchSource:0}: Error finding container 09c719d628bc52fbfaaa0f0aea422c95821d690f5e43923b49daf7d1b2107b9c: Status 404 returned error can't find the container with id 09c719d628bc52fbfaaa0f0aea422c95821d690f5e43923b49daf7d1b2107b9c
Sep 16 18:23:39.553343 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:39.553306    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978-tls-certs\") pod \"model-serving-api-54d89bd5fc-rxrjz\" (UID: \"d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978\") " pod="opendatahub/model-serving-api-54d89bd5fc-rxrjz"
Sep 16 18:23:39.555799 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:39.555776    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978-tls-certs\") pod \"model-serving-api-54d89bd5fc-rxrjz\" (UID: \"d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978\") " pod="opendatahub/model-serving-api-54d89bd5fc-rxrjz"
Sep 16 18:23:39.739418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:39.739389    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mmq55" event={"ID":"5c40361f-f58e-4143-80d6-c2a70e17acdb","Type":"ContainerStarted","Data":"09c719d628bc52fbfaaa0f0aea422c95821d690f5e43923b49daf7d1b2107b9c"}
Sep 16 18:23:39.796960 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:39.796917    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-rxrjz"
Sep 16 18:23:40.074857 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:40.074819    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rxrjz"]
Sep 16 18:23:40.077576 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:23:40.077547    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41bbcd3_42ce_48d6_b7fa_fb51e4fdd978.slice/crio-116381a8971a82d86fcf0a68c5b79d6b6d2adfdaa1074e2ebc4aed2de9d80604 WatchSource:0}: Error finding container 116381a8971a82d86fcf0a68c5b79d6b6d2adfdaa1074e2ebc4aed2de9d80604: Status 404 returned error can't find the container with id 116381a8971a82d86fcf0a68c5b79d6b6d2adfdaa1074e2ebc4aed2de9d80604
Sep 16 18:23:40.748397 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:40.748356    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-rxrjz" event={"ID":"d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978","Type":"ContainerStarted","Data":"116381a8971a82d86fcf0a68c5b79d6b6d2adfdaa1074e2ebc4aed2de9d80604"}
Sep 16 18:23:43.766076 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:43.766044    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mmq55" event={"ID":"5c40361f-f58e-4143-80d6-c2a70e17acdb","Type":"ContainerStarted","Data":"bc78b6189d6f726809b7e4f0e24b69d7bde290ecba97b2185b57588cd7f40c0f"}
Sep 16 18:23:43.766563 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:43.766181    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mmq55"
Sep 16 18:23:43.767586 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:43.767563    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-rxrjz" event={"ID":"d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978","Type":"ContainerStarted","Data":"d7dc95cdd3ae57c083c8a8a450ce836fba0fad5f8cd211176f5e00186e85f1e4"}
Sep 16 18:23:43.767871 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:43.767679    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-rxrjz"
Sep 16 18:23:43.783523 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:43.783440    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mmq55" podStartSLOduration=1.6223838860000002 podStartE2EDuration="5.783426136s" totalImagesPullingTime="4.16104225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:23:43.781459275 +0000 UTC m=+447.340627112" watchObservedRunningTime="2026-09-16 18:23:43.783426136 +0000 UTC m=+447.342593969"
Sep 16 18:23:43.798791 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:43.798747    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-rxrjz" podStartSLOduration=2.332295732 podStartE2EDuration="5.798730711s" totalImagesPullingTime="3.466434979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:23:43.79775216 +0000 UTC m=+447.356919992" watchObservedRunningTime="2026-09-16 18:23:43.798730711 +0000 UTC m=+447.357898682"
Sep 16 18:23:47.369382 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.369333    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"]
Sep 16 18:23:47.372441 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.372420    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"
Sep 16 18:23:47.376648 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.376624    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 18:23:47.376760 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.376675    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 18:23:47.378134 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.378116    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lg7l8\""
Sep 16 18:23:47.378259 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.378119    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 18:23:47.392446 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.392408    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"]
Sep 16 18:23:47.421734 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.421706    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dccl\" (UniqueName: \"kubernetes.io/projected/63b5c5d7-959c-4ca1-ac14-10074310bcaf-kube-api-access-5dccl\") pod \"lws-controller-manager-6fb494f6fb-hkmhl\" (UID: \"63b5c5d7-959c-4ca1-ac14-10074310bcaf\") " pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"
Sep 16 18:23:47.421878 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.421749    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63b5c5d7-959c-4ca1-ac14-10074310bcaf-cert\") pod \"lws-controller-manager-6fb494f6fb-hkmhl\" (UID: \"63b5c5d7-959c-4ca1-ac14-10074310bcaf\") " pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"
Sep 16 18:23:47.421878 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.421771    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63b5c5d7-959c-4ca1-ac14-10074310bcaf-metrics-cert\") pod \"lws-controller-manager-6fb494f6fb-hkmhl\" (UID: \"63b5c5d7-959c-4ca1-ac14-10074310bcaf\") " pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"
Sep 16 18:23:47.421878 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.421851    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/63b5c5d7-959c-4ca1-ac14-10074310bcaf-manager-config\") pod \"lws-controller-manager-6fb494f6fb-hkmhl\" (UID: \"63b5c5d7-959c-4ca1-ac14-10074310bcaf\") " pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"
Sep 16 18:23:47.523072 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.523039    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dccl\" (UniqueName: \"kubernetes.io/projected/63b5c5d7-959c-4ca1-ac14-10074310bcaf-kube-api-access-5dccl\") pod \"lws-controller-manager-6fb494f6fb-hkmhl\" (UID: \"63b5c5d7-959c-4ca1-ac14-10074310bcaf\") " pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"
Sep 16 18:23:47.523284 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.523085    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63b5c5d7-959c-4ca1-ac14-10074310bcaf-cert\") pod \"lws-controller-manager-6fb494f6fb-hkmhl\" (UID: \"63b5c5d7-959c-4ca1-ac14-10074310bcaf\") " pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"
Sep 16 18:23:47.523284 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.523110    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63b5c5d7-959c-4ca1-ac14-10074310bcaf-metrics-cert\") pod \"lws-controller-manager-6fb494f6fb-hkmhl\" (UID: \"63b5c5d7-959c-4ca1-ac14-10074310bcaf\") " pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"
Sep 16 18:23:47.523284 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.523139    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/63b5c5d7-959c-4ca1-ac14-10074310bcaf-manager-config\") pod \"lws-controller-manager-6fb494f6fb-hkmhl\" (UID: \"63b5c5d7-959c-4ca1-ac14-10074310bcaf\") " pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"
Sep 16 18:23:47.523770 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.523736    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/63b5c5d7-959c-4ca1-ac14-10074310bcaf-manager-config\") pod \"lws-controller-manager-6fb494f6fb-hkmhl\" (UID: \"63b5c5d7-959c-4ca1-ac14-10074310bcaf\") " pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"
Sep 16 18:23:47.525629 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.525597    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63b5c5d7-959c-4ca1-ac14-10074310bcaf-cert\") pod \"lws-controller-manager-6fb494f6fb-hkmhl\" (UID: \"63b5c5d7-959c-4ca1-ac14-10074310bcaf\") " pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"
Sep 16 18:23:47.525740 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.525672    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63b5c5d7-959c-4ca1-ac14-10074310bcaf-metrics-cert\") pod \"lws-controller-manager-6fb494f6fb-hkmhl\" (UID: \"63b5c5d7-959c-4ca1-ac14-10074310bcaf\") " pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"
Sep 16 18:23:47.539637 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.539615    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dccl\" (UniqueName: \"kubernetes.io/projected/63b5c5d7-959c-4ca1-ac14-10074310bcaf-kube-api-access-5dccl\") pod \"lws-controller-manager-6fb494f6fb-hkmhl\" (UID: \"63b5c5d7-959c-4ca1-ac14-10074310bcaf\") " pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"
Sep 16 18:23:47.681670 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.681590    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"
Sep 16 18:23:47.979367 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:47.979336    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"]
Sep 16 18:23:47.982829 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:23:47.982804    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b5c5d7_959c_4ca1_ac14_10074310bcaf.slice/crio-c67d77ec4e7bc875f6a92c86b9b67665283cf36fb7675f7e1f7f51ef16d9fe70 WatchSource:0}: Error finding container c67d77ec4e7bc875f6a92c86b9b67665283cf36fb7675f7e1f7f51ef16d9fe70: Status 404 returned error can't find the container with id c67d77ec4e7bc875f6a92c86b9b67665283cf36fb7675f7e1f7f51ef16d9fe70
Sep 16 18:23:48.789993 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:48.789957    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl" event={"ID":"63b5c5d7-959c-4ca1-ac14-10074310bcaf","Type":"ContainerStarted","Data":"c67d77ec4e7bc875f6a92c86b9b67665283cf36fb7675f7e1f7f51ef16d9fe70"}
Sep 16 18:23:49.095805 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.095732    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ls77m"]
Sep 16 18:23:49.099625 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.099603    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ls77m"
Sep 16 18:23:49.103754 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.103735    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-k2bp4\""
Sep 16 18:23:49.104342 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.104323    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 16 18:23:49.115344 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.115317    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ls77m"]
Sep 16 18:23:49.125281 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.125258    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q8q9l"]
Sep 16 18:23:49.129954 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.129931    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q8q9l"
Sep 16 18:23:49.133583 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.133558    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cc1f4df-c6ea-4a32-ba5e-d442c47e2665-cert\") pod \"kserve-controller-manager-d6f4b65b8-ls77m\" (UID: \"6cc1f4df-c6ea-4a32-ba5e-d442c47e2665\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ls77m"
Sep 16 18:23:49.133701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.133617    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pwz2\" (UniqueName: \"kubernetes.io/projected/6cc1f4df-c6ea-4a32-ba5e-d442c47e2665-kube-api-access-5pwz2\") pod \"kserve-controller-manager-d6f4b65b8-ls77m\" (UID: \"6cc1f4df-c6ea-4a32-ba5e-d442c47e2665\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ls77m"
Sep 16 18:23:49.133980 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.133959    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 16 18:23:49.134098 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.134059    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7dl9w\""
Sep 16 18:23:49.145350 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.145266    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q8q9l"]
Sep 16 18:23:49.234546 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.234515    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pwz2\" (UniqueName: \"kubernetes.io/projected/6cc1f4df-c6ea-4a32-ba5e-d442c47e2665-kube-api-access-5pwz2\") pod \"kserve-controller-manager-d6f4b65b8-ls77m\" (UID: \"6cc1f4df-c6ea-4a32-ba5e-d442c47e2665\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ls77m"
Sep 16 18:23:49.234708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.234577    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8rkp\" (UniqueName: \"kubernetes.io/projected/adf3b276-9306-4c42-827d-e08851a774b6-kube-api-access-q8rkp\") pod \"llmisvc-controller-manager-777c6bd7d5-q8q9l\" (UID: \"adf3b276-9306-4c42-827d-e08851a774b6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q8q9l"
Sep 16 18:23:49.234708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.234611    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adf3b276-9306-4c42-827d-e08851a774b6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q8q9l\" (UID: \"adf3b276-9306-4c42-827d-e08851a774b6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q8q9l"
Sep 16 18:23:49.234708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.234679    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cc1f4df-c6ea-4a32-ba5e-d442c47e2665-cert\") pod \"kserve-controller-manager-d6f4b65b8-ls77m\" (UID: \"6cc1f4df-c6ea-4a32-ba5e-d442c47e2665\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ls77m"
Sep 16 18:23:49.234822 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:23:49.234797    2554 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 16 18:23:49.234856 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:23:49.234850    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cc1f4df-c6ea-4a32-ba5e-d442c47e2665-cert podName:6cc1f4df-c6ea-4a32-ba5e-d442c47e2665 nodeName:}" failed. No retries permitted until 2026-09-16 18:23:49.734832763 +0000 UTC m=+453.294000594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6cc1f4df-c6ea-4a32-ba5e-d442c47e2665-cert") pod "kserve-controller-manager-d6f4b65b8-ls77m" (UID: "6cc1f4df-c6ea-4a32-ba5e-d442c47e2665") : secret "kserve-webhook-server-cert" not found
Sep 16 18:23:49.269949 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.269917    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pwz2\" (UniqueName: \"kubernetes.io/projected/6cc1f4df-c6ea-4a32-ba5e-d442c47e2665-kube-api-access-5pwz2\") pod \"kserve-controller-manager-d6f4b65b8-ls77m\" (UID: \"6cc1f4df-c6ea-4a32-ba5e-d442c47e2665\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ls77m"
Sep 16 18:23:49.336068 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.336030    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8rkp\" (UniqueName: \"kubernetes.io/projected/adf3b276-9306-4c42-827d-e08851a774b6-kube-api-access-q8rkp\") pod \"llmisvc-controller-manager-777c6bd7d5-q8q9l\" (UID: \"adf3b276-9306-4c42-827d-e08851a774b6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q8q9l"
Sep 16 18:23:49.336282 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.336077    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adf3b276-9306-4c42-827d-e08851a774b6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q8q9l\" (UID: \"adf3b276-9306-4c42-827d-e08851a774b6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q8q9l"
Sep 16 18:23:49.338717 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.338690    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adf3b276-9306-4c42-827d-e08851a774b6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q8q9l\" (UID: \"adf3b276-9306-4c42-827d-e08851a774b6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q8q9l"
Sep 16 18:23:49.348985 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.348919    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8rkp\" (UniqueName: \"kubernetes.io/projected/adf3b276-9306-4c42-827d-e08851a774b6-kube-api-access-q8rkp\") pod \"llmisvc-controller-manager-777c6bd7d5-q8q9l\" (UID: \"adf3b276-9306-4c42-827d-e08851a774b6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q8q9l"
Sep 16 18:23:49.443518 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.443482    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q8q9l"
Sep 16 18:23:49.728075 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.728048    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q8q9l"]
Sep 16 18:23:49.740214 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:49.740188    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cc1f4df-c6ea-4a32-ba5e-d442c47e2665-cert\") pod \"kserve-controller-manager-d6f4b65b8-ls77m\" (UID: \"6cc1f4df-c6ea-4a32-ba5e-d442c47e2665\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ls77m"
Sep 16 18:23:49.740360 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:23:49.740340    2554 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 16 18:23:49.740437 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:23:49.740425    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cc1f4df-c6ea-4a32-ba5e-d442c47e2665-cert podName:6cc1f4df-c6ea-4a32-ba5e-d442c47e2665 nodeName:}" failed. No retries permitted until 2026-09-16 18:23:50.740392469 +0000 UTC m=+454.299560302 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6cc1f4df-c6ea-4a32-ba5e-d442c47e2665-cert") pod "kserve-controller-manager-d6f4b65b8-ls77m" (UID: "6cc1f4df-c6ea-4a32-ba5e-d442c47e2665") : secret "kserve-webhook-server-cert" not found
Sep 16 18:23:49.956627 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:23:49.956590    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podadf3b276_9306_4c42_827d_e08851a774b6.slice/crio-f6c8e423dc55907fd5b8601aa89e58aa79db14814a5f9ad0e090c6e361e13d2d WatchSource:0}: Error finding container f6c8e423dc55907fd5b8601aa89e58aa79db14814a5f9ad0e090c6e361e13d2d: Status 404 returned error can't find the container with id f6c8e423dc55907fd5b8601aa89e58aa79db14814a5f9ad0e090c6e361e13d2d
Sep 16 18:23:50.749906 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:50.749871    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cc1f4df-c6ea-4a32-ba5e-d442c47e2665-cert\") pod \"kserve-controller-manager-d6f4b65b8-ls77m\" (UID: \"6cc1f4df-c6ea-4a32-ba5e-d442c47e2665\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ls77m"
Sep 16 18:23:50.752210 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:50.752183    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cc1f4df-c6ea-4a32-ba5e-d442c47e2665-cert\") pod \"kserve-controller-manager-d6f4b65b8-ls77m\" (UID: \"6cc1f4df-c6ea-4a32-ba5e-d442c47e2665\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ls77m"
Sep 16 18:23:50.802531 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:50.802490    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q8q9l" event={"ID":"adf3b276-9306-4c42-827d-e08851a774b6","Type":"ContainerStarted","Data":"f6c8e423dc55907fd5b8601aa89e58aa79db14814a5f9ad0e090c6e361e13d2d"}
Sep 16 18:23:50.803823 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:50.803796    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl" event={"ID":"63b5c5d7-959c-4ca1-ac14-10074310bcaf","Type":"ContainerStarted","Data":"bbbc01dfcdb9b5b9d676d66d402114c9a76853869af002f4d0d7b526454d3d85"}
Sep 16 18:23:50.803953 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:50.803921    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"
Sep 16 18:23:50.882359 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:50.882299    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl" podStartSLOduration=1.848586408 podStartE2EDuration="3.88227899s" totalImagesPullingTime="2.033692582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:23:50.880963497 +0000 UTC m=+454.440131332" watchObservedRunningTime="2026-09-16 18:23:50.88227899 +0000 UTC m=+454.441446825"
Sep 16 18:23:50.911432 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:50.911404    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ls77m"
Sep 16 18:23:51.456126 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:51.455898    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ls77m"]
Sep 16 18:23:51.458906 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:23:51.458872    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc1f4df_c6ea_4a32_ba5e_d442c47e2665.slice/crio-02dadd120ce5bc7bdf2a0ae86fcca11f6f5de60a8d9814f00f74c6044e2c6ffe WatchSource:0}: Error finding container 02dadd120ce5bc7bdf2a0ae86fcca11f6f5de60a8d9814f00f74c6044e2c6ffe: Status 404 returned error can't find the container with id 02dadd120ce5bc7bdf2a0ae86fcca11f6f5de60a8d9814f00f74c6044e2c6ffe
Sep 16 18:23:51.809480 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:51.809445    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ls77m" event={"ID":"6cc1f4df-c6ea-4a32-ba5e-d442c47e2665","Type":"ContainerStarted","Data":"02dadd120ce5bc7bdf2a0ae86fcca11f6f5de60a8d9814f00f74c6044e2c6ffe"}
Sep 16 18:23:53.823658 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:53.823620    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q8q9l" event={"ID":"adf3b276-9306-4c42-827d-e08851a774b6","Type":"ContainerStarted","Data":"cd638375fa4fed7f43536d65bfa47b592bacf7f942a7a0338c3652fa96864c27"}
Sep 16 18:23:53.824023 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:53.823703    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q8q9l"
Sep 16 18:23:53.854759 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:53.854698    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q8q9l" podStartSLOduration=1.8277805599999999 podStartE2EDuration="4.854677073s" totalImagesPullingTime="3.026896513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:23:53.854354843 +0000 UTC m=+457.413522677" watchObservedRunningTime="2026-09-16 18:23:53.854677073 +0000 UTC m=+457.413844908"
Sep 16 18:23:54.773838 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:54.773807    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mmq55"
Sep 16 18:23:54.775695 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:54.775673    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-rxrjz"
Sep 16 18:23:54.831056 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:54.831022    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ls77m" event={"ID":"6cc1f4df-c6ea-4a32-ba5e-d442c47e2665","Type":"ContainerStarted","Data":"204d4bdbcdd7e45efaf5b1d7c75d4d34a30519f2d20b09ee37761464bc1a9b44"}
Sep 16 18:23:54.831539 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:54.831154    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ls77m"
Sep 16 18:23:54.883898 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:23:54.883851    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ls77m" podStartSLOduration=3.474093696 podStartE2EDuration="5.883831524s" totalImagesPullingTime="2.409737828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:23:54.883493116 +0000 UTC m=+458.442660949" watchObservedRunningTime="2026-09-16 18:23:54.883831524 +0000 UTC m=+458.442999359"
Sep 16 18:24:01.811532 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:24:01.811500    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6fb494f6fb-hkmhl"
Sep 16 18:24:24.833149 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:24:24.833116    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q8q9l"
Sep 16 18:24:25.839718 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:24:25.839685    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ls77m"
Sep 16 18:25:07.757058 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:07.757021    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"]
Sep 16 18:25:07.760393 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:07.760377    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"
Sep 16 18:25:07.762742 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:07.762721    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:25:07.762847 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:07.762753    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6bv26\""
Sep 16 18:25:07.762847 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:07.762780    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:25:07.775124 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:07.775100    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"]
Sep 16 18:25:07.897985 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:07.897941    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd5vk\" (UniqueName: \"kubernetes.io/projected/735dfabe-404f-4ea0-9782-59c9bd9d9bbc-kube-api-access-hd5vk\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vfpkj\" (UID: \"735dfabe-404f-4ea0-9782-59c9bd9d9bbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"
Sep 16 18:25:07.897985 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:07.897988    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/735dfabe-404f-4ea0-9782-59c9bd9d9bbc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vfpkj\" (UID: \"735dfabe-404f-4ea0-9782-59c9bd9d9bbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"
Sep 16 18:25:07.999354 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:07.999312    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/735dfabe-404f-4ea0-9782-59c9bd9d9bbc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vfpkj\" (UID: \"735dfabe-404f-4ea0-9782-59c9bd9d9bbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"
Sep 16 18:25:07.999544 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:07.999523    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd5vk\" (UniqueName: \"kubernetes.io/projected/735dfabe-404f-4ea0-9782-59c9bd9d9bbc-kube-api-access-hd5vk\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vfpkj\" (UID: \"735dfabe-404f-4ea0-9782-59c9bd9d9bbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"
Sep 16 18:25:07.999789 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:07.999768    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/735dfabe-404f-4ea0-9782-59c9bd9d9bbc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vfpkj\" (UID: \"735dfabe-404f-4ea0-9782-59c9bd9d9bbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"
Sep 16 18:25:08.019319 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:08.019269    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd5vk\" (UniqueName: \"kubernetes.io/projected/735dfabe-404f-4ea0-9782-59c9bd9d9bbc-kube-api-access-hd5vk\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vfpkj\" (UID: \"735dfabe-404f-4ea0-9782-59c9bd9d9bbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"
Sep 16 18:25:08.069959 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:08.069916    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"
Sep 16 18:25:08.355220 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:08.355195    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"]
Sep 16 18:25:08.357478 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:25:08.357453    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735dfabe_404f_4ea0_9782_59c9bd9d9bbc.slice/crio-6e018ba60024ddda23d3846e12a8d59a7ae181a48ec26b70145edaf2c2daa128 WatchSource:0}: Error finding container 6e018ba60024ddda23d3846e12a8d59a7ae181a48ec26b70145edaf2c2daa128: Status 404 returned error can't find the container with id 6e018ba60024ddda23d3846e12a8d59a7ae181a48ec26b70145edaf2c2daa128
Sep 16 18:25:09.159677 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:09.159605    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj" event={"ID":"735dfabe-404f-4ea0-9782-59c9bd9d9bbc","Type":"ContainerStarted","Data":"6e018ba60024ddda23d3846e12a8d59a7ae181a48ec26b70145edaf2c2daa128"}
Sep 16 18:25:12.000138 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:12.000104    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vjxkp"]
Sep 16 18:25:12.003415 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:12.003394    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vjxkp"
Sep 16 18:25:12.005752 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:12.005735    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 18:25:12.005832 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:12.005770    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cb9x6\""
Sep 16 18:25:12.023204 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:12.023180    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vjxkp"]
Sep 16 18:25:12.136512 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:12.136476    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtqnn\" (UniqueName: \"kubernetes.io/projected/0ec2d7cb-90f5-410b-8886-ce49c5accdc0-kube-api-access-dtqnn\") pod \"dns-operator-controller-manager-7b84cd7c9c-vjxkp\" (UID: \"0ec2d7cb-90f5-410b-8886-ce49c5accdc0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vjxkp"
Sep 16 18:25:12.237786 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:12.237755    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtqnn\" (UniqueName: \"kubernetes.io/projected/0ec2d7cb-90f5-410b-8886-ce49c5accdc0-kube-api-access-dtqnn\") pod \"dns-operator-controller-manager-7b84cd7c9c-vjxkp\" (UID: \"0ec2d7cb-90f5-410b-8886-ce49c5accdc0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vjxkp"
Sep 16 18:25:12.245857 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:12.245834    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtqnn\" (UniqueName: \"kubernetes.io/projected/0ec2d7cb-90f5-410b-8886-ce49c5accdc0-kube-api-access-dtqnn\") pod \"dns-operator-controller-manager-7b84cd7c9c-vjxkp\" (UID: \"0ec2d7cb-90f5-410b-8886-ce49c5accdc0\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vjxkp"
Sep 16 18:25:12.313139 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:12.313085    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vjxkp"
Sep 16 18:25:12.589352 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:12.589327    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vjxkp"]
Sep 16 18:25:12.591767 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:25:12.591736    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec2d7cb_90f5_410b_8886_ce49c5accdc0.slice/crio-1740806390a4cc88aed88662f0b9252b46305d59d3ab67db1508324d5286efd3 WatchSource:0}: Error finding container 1740806390a4cc88aed88662f0b9252b46305d59d3ab67db1508324d5286efd3: Status 404 returned error can't find the container with id 1740806390a4cc88aed88662f0b9252b46305d59d3ab67db1508324d5286efd3
Sep 16 18:25:13.177183 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:13.177144    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vjxkp" event={"ID":"0ec2d7cb-90f5-410b-8886-ce49c5accdc0","Type":"ContainerStarted","Data":"1740806390a4cc88aed88662f0b9252b46305d59d3ab67db1508324d5286efd3"}
Sep 16 18:25:16.196576 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:16.196533    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vjxkp" event={"ID":"0ec2d7cb-90f5-410b-8886-ce49c5accdc0","Type":"ContainerStarted","Data":"053e3e5fd935a376d3d3930cdb2b2086cc14d8d0172dd4764f9e808dcef0a7ac"}
Sep 16 18:25:16.197029 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:16.196692    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vjxkp"
Sep 16 18:25:16.215658 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:16.215617    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vjxkp" podStartSLOduration=2.382154447 podStartE2EDuration="5.215596781s" totalImagesPullingTime="2.833442334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:25:16.212860805 +0000 UTC m=+539.772028641" watchObservedRunningTime="2026-09-16 18:25:16.215596781 +0000 UTC m=+539.774764615"
Sep 16 18:25:19.213978 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:19.213940    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj" event={"ID":"735dfabe-404f-4ea0-9782-59c9bd9d9bbc","Type":"ContainerStarted","Data":"0c46d84ea5c5cb815be5a9fb09dd631286bf05eff99e0afa154767c8e75d81bf"}
Sep 16 18:25:19.214371 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:19.214073    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"
Sep 16 18:25:19.235163 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:19.234657    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj" podStartSLOduration=1.938549426 podStartE2EDuration="12.234637853s" totalImagesPullingTime="10.296088427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:25:19.232646158 +0000 UTC m=+542.791813992" watchObservedRunningTime="2026-09-16 18:25:19.234637853 +0000 UTC m=+542.793805688"
Sep 16 18:25:27.204514 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:27.204476    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vjxkp"
Sep 16 18:25:30.221172 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:30.221139    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"
Sep 16 18:25:31.351245 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:31.351198    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bptqv"]
Sep 16 18:25:31.357673 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:31.357647    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bptqv"
Sep 16 18:25:31.368925 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:31.368899    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bptqv"]
Sep 16 18:25:31.402753 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:31.402711    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c8994ce-5edb-412f-a84c-934400efb033-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-bptqv\" (UID: \"0c8994ce-5edb-412f-a84c-934400efb033\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bptqv"
Sep 16 18:25:31.403017 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:31.402760    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjb6n\" (UniqueName: \"kubernetes.io/projected/0c8994ce-5edb-412f-a84c-934400efb033-kube-api-access-wjb6n\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-bptqv\" (UID: \"0c8994ce-5edb-412f-a84c-934400efb033\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bptqv"
Sep 16 18:25:31.459901 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:31.459862    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bptqv"]
Sep 16 18:25:31.460209 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:25:31.460187    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-wjb6n], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bptqv" podUID="0c8994ce-5edb-412f-a84c-934400efb033"
Sep 16 18:25:31.503702 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:31.503652    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c8994ce-5edb-412f-a84c-934400efb033-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-bptqv\" (UID: \"0c8994ce-5edb-412f-a84c-934400efb033\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bptqv"
Sep 16 18:25:31.504457 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:31.504429    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjb6n\" (UniqueName: \"kubernetes.io/projected/0c8994ce-5edb-412f-a84c-934400efb033-kube-api-access-wjb6n\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-bptqv\" (UID: \"0c8994ce-5edb-412f-a84c-934400efb033\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bptqv"
Sep 16 18:25:31.504630 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:31.504365    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c8994ce-5edb-412f-a84c-934400efb033-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-bptqv\" (UID: \"0c8994ce-5edb-412f-a84c-934400efb033\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bptqv"
Sep 16 18:25:31.515701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:31.515667    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjb6n\" (UniqueName: \"kubernetes.io/projected/0c8994ce-5edb-412f-a84c-934400efb033-kube-api-access-wjb6n\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-bptqv\" (UID: \"0c8994ce-5edb-412f-a84c-934400efb033\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bptqv"
Sep 16 18:25:32.274416 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.274382    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bptqv"
Sep 16 18:25:32.281952 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.281926    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bptqv"
Sep 16 18:25:32.312738 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.312697    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bptqv"]
Sep 16 18:25:32.312962 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.312941    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c8994ce-5edb-412f-a84c-934400efb033-extensions-socket-volume\") pod \"0c8994ce-5edb-412f-a84c-934400efb033\" (UID: \"0c8994ce-5edb-412f-a84c-934400efb033\") "
Sep 16 18:25:32.313035 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.313009    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjb6n\" (UniqueName: \"kubernetes.io/projected/0c8994ce-5edb-412f-a84c-934400efb033-kube-api-access-wjb6n\") pod \"0c8994ce-5edb-412f-a84c-934400efb033\" (UID: \"0c8994ce-5edb-412f-a84c-934400efb033\") "
Sep 16 18:25:32.313272 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.313220    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c8994ce-5edb-412f-a84c-934400efb033-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0c8994ce-5edb-412f-a84c-934400efb033" (UID: "0c8994ce-5edb-412f-a84c-934400efb033"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:25:32.313385 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.313365    2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c8994ce-5edb-412f-a84c-934400efb033-extensions-socket-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:25:32.315248 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.315209    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8994ce-5edb-412f-a84c-934400efb033-kube-api-access-wjb6n" (OuterVolumeSpecName: "kube-api-access-wjb6n") pod "0c8994ce-5edb-412f-a84c-934400efb033" (UID: "0c8994ce-5edb-412f-a84c-934400efb033"). InnerVolumeSpecName "kube-api-access-wjb6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:25:32.325975 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.325942    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bptqv"]
Sep 16 18:25:32.344196 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.344160    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"]
Sep 16 18:25:32.344377 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.344354    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj" podUID="735dfabe-404f-4ea0-9782-59c9bd9d9bbc" containerName="manager" containerID="cri-o://0c46d84ea5c5cb815be5a9fb09dd631286bf05eff99e0afa154767c8e75d81bf" gracePeriod=2
Sep 16 18:25:32.362255 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.362205    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"]
Sep 16 18:25:32.376429 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.376398    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"]
Sep 16 18:25:32.376801 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.376787    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="735dfabe-404f-4ea0-9782-59c9bd9d9bbc" containerName="manager"
Sep 16 18:25:32.376847 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.376803    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="735dfabe-404f-4ea0-9782-59c9bd9d9bbc" containerName="manager"
Sep 16 18:25:32.376891 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.376865    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="735dfabe-404f-4ea0-9782-59c9bd9d9bbc" containerName="manager"
Sep 16 18:25:32.380187 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.380166    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"
Sep 16 18:25:32.382460 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:25:32.382418    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-vfpkj\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" podUID="735dfabe-404f-4ea0-9782-59c9bd9d9bbc" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"
Sep 16 18:25:32.395922 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.395887    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"]
Sep 16 18:25:32.414562 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.414529    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsq2t\" (UniqueName: \"kubernetes.io/projected/2044ffe7-e49c-4395-9274-28559326c470-kube-api-access-hsq2t\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7dnsj\" (UID: \"2044ffe7-e49c-4395-9274-28559326c470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"
Sep 16 18:25:32.414747 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.414575    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2044ffe7-e49c-4395-9274-28559326c470-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7dnsj\" (UID: \"2044ffe7-e49c-4395-9274-28559326c470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"
Sep 16 18:25:32.414747 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.414720    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjb6n\" (UniqueName: \"kubernetes.io/projected/0c8994ce-5edb-412f-a84c-934400efb033-kube-api-access-wjb6n\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:25:32.515557 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.515492    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2044ffe7-e49c-4395-9274-28559326c470-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7dnsj\" (UID: \"2044ffe7-e49c-4395-9274-28559326c470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"
Sep 16 18:25:32.515705 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.515622    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsq2t\" (UniqueName: \"kubernetes.io/projected/2044ffe7-e49c-4395-9274-28559326c470-kube-api-access-hsq2t\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7dnsj\" (UID: \"2044ffe7-e49c-4395-9274-28559326c470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"
Sep 16 18:25:32.516023 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.515986    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2044ffe7-e49c-4395-9274-28559326c470-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7dnsj\" (UID: \"2044ffe7-e49c-4395-9274-28559326c470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"
Sep 16 18:25:32.533516 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.533443    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsq2t\" (UniqueName: \"kubernetes.io/projected/2044ffe7-e49c-4395-9274-28559326c470-kube-api-access-hsq2t\") pod \"kuadrant-operator-controller-manager-76f8f686bf-7dnsj\" (UID: \"2044ffe7-e49c-4395-9274-28559326c470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"
Sep 16 18:25:32.582337 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.582304    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"
Sep 16 18:25:32.584420 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:25:32.584386    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-vfpkj\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" podUID="735dfabe-404f-4ea0-9782-59c9bd9d9bbc" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"
Sep 16 18:25:32.616884 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.616851    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd5vk\" (UniqueName: \"kubernetes.io/projected/735dfabe-404f-4ea0-9782-59c9bd9d9bbc-kube-api-access-hd5vk\") pod \"735dfabe-404f-4ea0-9782-59c9bd9d9bbc\" (UID: \"735dfabe-404f-4ea0-9782-59c9bd9d9bbc\") "
Sep 16 18:25:32.617050 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.616902    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/735dfabe-404f-4ea0-9782-59c9bd9d9bbc-extensions-socket-volume\") pod \"735dfabe-404f-4ea0-9782-59c9bd9d9bbc\" (UID: \"735dfabe-404f-4ea0-9782-59c9bd9d9bbc\") "
Sep 16 18:25:32.617440 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.617414    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735dfabe-404f-4ea0-9782-59c9bd9d9bbc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "735dfabe-404f-4ea0-9782-59c9bd9d9bbc" (UID: "735dfabe-404f-4ea0-9782-59c9bd9d9bbc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:25:32.619051 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.619030    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735dfabe-404f-4ea0-9782-59c9bd9d9bbc-kube-api-access-hd5vk" (OuterVolumeSpecName: "kube-api-access-hd5vk") pod "735dfabe-404f-4ea0-9782-59c9bd9d9bbc" (UID: "735dfabe-404f-4ea0-9782-59c9bd9d9bbc"). InnerVolumeSpecName "kube-api-access-hd5vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:25:32.718351 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.718318    2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/735dfabe-404f-4ea0-9782-59c9bd9d9bbc-extensions-socket-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:25:32.718351 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.718347    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd5vk\" (UniqueName: \"kubernetes.io/projected/735dfabe-404f-4ea0-9782-59c9bd9d9bbc-kube-api-access-hd5vk\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:25:32.725170 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.725143    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"
Sep 16 18:25:32.883810 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.883733    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c8994ce-5edb-412f-a84c-934400efb033" path="/var/lib/kubelet/pods/0c8994ce-5edb-412f-a84c-934400efb033/volumes"
Sep 16 18:25:32.884012 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:32.884000    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="735dfabe-404f-4ea0-9782-59c9bd9d9bbc" path="/var/lib/kubelet/pods/735dfabe-404f-4ea0-9782-59c9bd9d9bbc/volumes"
Sep 16 18:25:33.015122 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:33.015099    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"]
Sep 16 18:25:33.016868 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:25:33.016844    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2044ffe7_e49c_4395_9274_28559326c470.slice/crio-198c843ccea9e56616e99ed3148ebcb8f4e43d900752ba665336f4fcd5dec836 WatchSource:0}: Error finding container 198c843ccea9e56616e99ed3148ebcb8f4e43d900752ba665336f4fcd5dec836: Status 404 returned error can't find the container with id 198c843ccea9e56616e99ed3148ebcb8f4e43d900752ba665336f4fcd5dec836
Sep 16 18:25:33.281204 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:33.281169    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj" event={"ID":"2044ffe7-e49c-4395-9274-28559326c470","Type":"ContainerStarted","Data":"84b2a5b7b4ed761452ab7aaa94527fc1024f8b85c097079ecb299543ab651235"}
Sep 16 18:25:33.281424 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:33.281222    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj" event={"ID":"2044ffe7-e49c-4395-9274-28559326c470","Type":"ContainerStarted","Data":"198c843ccea9e56616e99ed3148ebcb8f4e43d900752ba665336f4fcd5dec836"}
Sep 16 18:25:33.281424 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:33.281340    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"
Sep 16 18:25:33.282619 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:33.282594    2554 generic.go:356] "Generic (PLEG): container finished" podID="735dfabe-404f-4ea0-9782-59c9bd9d9bbc" containerID="0c46d84ea5c5cb815be5a9fb09dd631286bf05eff99e0afa154767c8e75d81bf" exitCode=0
Sep 16 18:25:33.282704 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:33.282629    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vfpkj"
Sep 16 18:25:33.282704 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:33.282684    2554 scope.go:117] "RemoveContainer" containerID="0c46d84ea5c5cb815be5a9fb09dd631286bf05eff99e0afa154767c8e75d81bf"
Sep 16 18:25:33.282768 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:33.282718    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-bptqv"
Sep 16 18:25:33.292753 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:33.292732    2554 scope.go:117] "RemoveContainer" containerID="0c46d84ea5c5cb815be5a9fb09dd631286bf05eff99e0afa154767c8e75d81bf"
Sep 16 18:25:33.293479 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:25:33.293449    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c46d84ea5c5cb815be5a9fb09dd631286bf05eff99e0afa154767c8e75d81bf\": container with ID starting with 0c46d84ea5c5cb815be5a9fb09dd631286bf05eff99e0afa154767c8e75d81bf not found: ID does not exist" containerID="0c46d84ea5c5cb815be5a9fb09dd631286bf05eff99e0afa154767c8e75d81bf"
Sep 16 18:25:33.293564 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:33.293488    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c46d84ea5c5cb815be5a9fb09dd631286bf05eff99e0afa154767c8e75d81bf"} err="failed to get container status \"0c46d84ea5c5cb815be5a9fb09dd631286bf05eff99e0afa154767c8e75d81bf\": rpc error: code = NotFound desc = could not find container \"0c46d84ea5c5cb815be5a9fb09dd631286bf05eff99e0afa154767c8e75d81bf\": container with ID starting with 0c46d84ea5c5cb815be5a9fb09dd631286bf05eff99e0afa154767c8e75d81bf not found: ID does not exist"
Sep 16 18:25:33.312989 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:33.312403    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj" podStartSLOduration=1.31238166 podStartE2EDuration="1.31238166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:25:33.311696428 +0000 UTC m=+556.870864256" watchObservedRunningTime="2026-09-16 18:25:33.31238166 +0000 UTC m=+556.871549495"
Sep 16 18:25:44.289753 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:44.289725    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"
Sep 16 18:25:47.326905 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.326863    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"]
Sep 16 18:25:47.327423 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.327050    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj" podUID="2044ffe7-e49c-4395-9274-28559326c470" containerName="manager" containerID="cri-o://84b2a5b7b4ed761452ab7aaa94527fc1024f8b85c097079ecb299543ab651235" gracePeriod=10
Sep 16 18:25:47.569850 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.569825    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"
Sep 16 18:25:47.590423 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.590373    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"]
Sep 16 18:25:47.590756 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.590741    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2044ffe7-e49c-4395-9274-28559326c470" containerName="manager"
Sep 16 18:25:47.590809 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.590757    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2044ffe7-e49c-4395-9274-28559326c470" containerName="manager"
Sep 16 18:25:47.590844 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.590828    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2044ffe7-e49c-4395-9274-28559326c470" containerName="manager"
Sep 16 18:25:47.592561 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.592547    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"
Sep 16 18:25:47.608327 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.608303    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"]
Sep 16 18:25:47.653662 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.653637    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsq2t\" (UniqueName: \"kubernetes.io/projected/2044ffe7-e49c-4395-9274-28559326c470-kube-api-access-hsq2t\") pod \"2044ffe7-e49c-4395-9274-28559326c470\" (UID: \"2044ffe7-e49c-4395-9274-28559326c470\") "
Sep 16 18:25:47.653774 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.653679    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2044ffe7-e49c-4395-9274-28559326c470-extensions-socket-volume\") pod \"2044ffe7-e49c-4395-9274-28559326c470\" (UID: \"2044ffe7-e49c-4395-9274-28559326c470\") "
Sep 16 18:25:47.654068 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.654040    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2044ffe7-e49c-4395-9274-28559326c470-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2044ffe7-e49c-4395-9274-28559326c470" (UID: "2044ffe7-e49c-4395-9274-28559326c470"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:25:47.655668 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.655646    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2044ffe7-e49c-4395-9274-28559326c470-kube-api-access-hsq2t" (OuterVolumeSpecName: "kube-api-access-hsq2t") pod "2044ffe7-e49c-4395-9274-28559326c470" (UID: "2044ffe7-e49c-4395-9274-28559326c470"). InnerVolumeSpecName "kube-api-access-hsq2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:25:47.755050 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.755024    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b82qr\" (UniqueName: \"kubernetes.io/projected/f5313a4b-6151-4a66-9688-b4c6b29f3a95-kube-api-access-b82qr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pqhln\" (UID: \"f5313a4b-6151-4a66-9688-b4c6b29f3a95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"
Sep 16 18:25:47.755142 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.755068    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5313a4b-6151-4a66-9688-b4c6b29f3a95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pqhln\" (UID: \"f5313a4b-6151-4a66-9688-b4c6b29f3a95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"
Sep 16 18:25:47.755142 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.755099    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsq2t\" (UniqueName: \"kubernetes.io/projected/2044ffe7-e49c-4395-9274-28559326c470-kube-api-access-hsq2t\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:25:47.755142 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.755110    2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2044ffe7-e49c-4395-9274-28559326c470-extensions-socket-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:25:47.856113 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.856032    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b82qr\" (UniqueName: \"kubernetes.io/projected/f5313a4b-6151-4a66-9688-b4c6b29f3a95-kube-api-access-b82qr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pqhln\" (UID: \"f5313a4b-6151-4a66-9688-b4c6b29f3a95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"
Sep 16 18:25:47.856113 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.856086    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5313a4b-6151-4a66-9688-b4c6b29f3a95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pqhln\" (UID: \"f5313a4b-6151-4a66-9688-b4c6b29f3a95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"
Sep 16 18:25:47.856510 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.856491    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5313a4b-6151-4a66-9688-b4c6b29f3a95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pqhln\" (UID: \"f5313a4b-6151-4a66-9688-b4c6b29f3a95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"
Sep 16 18:25:47.864868 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.864843    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b82qr\" (UniqueName: \"kubernetes.io/projected/f5313a4b-6151-4a66-9688-b4c6b29f3a95-kube-api-access-b82qr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-pqhln\" (UID: \"f5313a4b-6151-4a66-9688-b4c6b29f3a95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"
Sep 16 18:25:47.904968 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:47.904934    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"
Sep 16 18:25:48.179958 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:48.179770    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"]
Sep 16 18:25:48.182882 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:25:48.182857    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5313a4b_6151_4a66_9688_b4c6b29f3a95.slice/crio-66c9ef183e286738b5949ab4e8c84ec375e328186352ca2ca6ec3d8f5e6ea895 WatchSource:0}: Error finding container 66c9ef183e286738b5949ab4e8c84ec375e328186352ca2ca6ec3d8f5e6ea895: Status 404 returned error can't find the container with id 66c9ef183e286738b5949ab4e8c84ec375e328186352ca2ca6ec3d8f5e6ea895
Sep 16 18:25:48.356347 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:48.356314    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln" event={"ID":"f5313a4b-6151-4a66-9688-b4c6b29f3a95","Type":"ContainerStarted","Data":"18ac9806dc04f192cf34ccfeced8bae480a03a74837d54c22d2f4249ac0aeb1b"}
Sep 16 18:25:48.356763 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:48.356366    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln" event={"ID":"f5313a4b-6151-4a66-9688-b4c6b29f3a95","Type":"ContainerStarted","Data":"66c9ef183e286738b5949ab4e8c84ec375e328186352ca2ca6ec3d8f5e6ea895"}
Sep 16 18:25:48.356763 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:48.356498    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"
Sep 16 18:25:48.357720 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:48.357699    2554 generic.go:356] "Generic (PLEG): container finished" podID="2044ffe7-e49c-4395-9274-28559326c470" containerID="84b2a5b7b4ed761452ab7aaa94527fc1024f8b85c097079ecb299543ab651235" exitCode=0
Sep 16 18:25:48.357777 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:48.357732    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"
Sep 16 18:25:48.357833 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:48.357781    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj" event={"ID":"2044ffe7-e49c-4395-9274-28559326c470","Type":"ContainerDied","Data":"84b2a5b7b4ed761452ab7aaa94527fc1024f8b85c097079ecb299543ab651235"}
Sep 16 18:25:48.357890 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:48.357832    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj" event={"ID":"2044ffe7-e49c-4395-9274-28559326c470","Type":"ContainerDied","Data":"198c843ccea9e56616e99ed3148ebcb8f4e43d900752ba665336f4fcd5dec836"}
Sep 16 18:25:48.357890 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:48.357865    2554 scope.go:117] "RemoveContainer" containerID="84b2a5b7b4ed761452ab7aaa94527fc1024f8b85c097079ecb299543ab651235"
Sep 16 18:25:48.366634 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:48.366613    2554 scope.go:117] "RemoveContainer" containerID="84b2a5b7b4ed761452ab7aaa94527fc1024f8b85c097079ecb299543ab651235"
Sep 16 18:25:48.366928 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:25:48.366911    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b2a5b7b4ed761452ab7aaa94527fc1024f8b85c097079ecb299543ab651235\": container with ID starting with 84b2a5b7b4ed761452ab7aaa94527fc1024f8b85c097079ecb299543ab651235 not found: ID does not exist" containerID="84b2a5b7b4ed761452ab7aaa94527fc1024f8b85c097079ecb299543ab651235"
Sep 16 18:25:48.366990 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:48.366938    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b2a5b7b4ed761452ab7aaa94527fc1024f8b85c097079ecb299543ab651235"} err="failed to get container status \"84b2a5b7b4ed761452ab7aaa94527fc1024f8b85c097079ecb299543ab651235\": rpc error: code = NotFound desc = could not find container \"84b2a5b7b4ed761452ab7aaa94527fc1024f8b85c097079ecb299543ab651235\": container with ID starting with 84b2a5b7b4ed761452ab7aaa94527fc1024f8b85c097079ecb299543ab651235 not found: ID does not exist"
Sep 16 18:25:48.376495 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:48.376444    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln" podStartSLOduration=1.376427178 podStartE2EDuration="1.376427178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:25:48.376390731 +0000 UTC m=+571.935558566" watchObservedRunningTime="2026-09-16 18:25:48.376427178 +0000 UTC m=+571.935595014"
Sep 16 18:25:48.390640 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:48.390615    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"]
Sep 16 18:25:48.394731 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:48.394711    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-7dnsj"]
Sep 16 18:25:48.884382 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:48.884347    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2044ffe7-e49c-4395-9274-28559326c470" path="/var/lib/kubelet/pods/2044ffe7-e49c-4395-9274-28559326c470/volumes"
Sep 16 18:25:59.367215 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:25:59.367186    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"
Sep 16 18:26:16.789656 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:16.789631    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:26:16.791191 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:16.791172    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:26:19.753373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:19.753338    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8jfjm"]
Sep 16 18:26:19.758044 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:19.758024    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-8jfjm"
Sep 16 18:26:19.760276 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:19.760255    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 18:26:19.760276 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:19.760270    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-57fd7\""
Sep 16 18:26:19.766432 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:19.766411    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8jfjm"]
Sep 16 18:26:19.824574 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:19.824548    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/654a521c-2473-4a47-a492-5e4ca7aad6b5-config-file\") pod \"limitador-limitador-8588749977-8jfjm\" (UID: \"654a521c-2473-4a47-a492-5e4ca7aad6b5\") " pod="kuadrant-system/limitador-limitador-8588749977-8jfjm"
Sep 16 18:26:19.824687 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:19.824667    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ntf\" (UniqueName: \"kubernetes.io/projected/654a521c-2473-4a47-a492-5e4ca7aad6b5-kube-api-access-68ntf\") pod \"limitador-limitador-8588749977-8jfjm\" (UID: \"654a521c-2473-4a47-a492-5e4ca7aad6b5\") " pod="kuadrant-system/limitador-limitador-8588749977-8jfjm"
Sep 16 18:26:19.848607 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:19.848576    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8jfjm"]
Sep 16 18:26:19.926059 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:19.926031    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/654a521c-2473-4a47-a492-5e4ca7aad6b5-config-file\") pod \"limitador-limitador-8588749977-8jfjm\" (UID: \"654a521c-2473-4a47-a492-5e4ca7aad6b5\") " pod="kuadrant-system/limitador-limitador-8588749977-8jfjm"
Sep 16 18:26:19.926626 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:19.926223    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ntf\" (UniqueName: \"kubernetes.io/projected/654a521c-2473-4a47-a492-5e4ca7aad6b5-kube-api-access-68ntf\") pod \"limitador-limitador-8588749977-8jfjm\" (UID: \"654a521c-2473-4a47-a492-5e4ca7aad6b5\") " pod="kuadrant-system/limitador-limitador-8588749977-8jfjm"
Sep 16 18:26:19.927044 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:19.927024    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/654a521c-2473-4a47-a492-5e4ca7aad6b5-config-file\") pod \"limitador-limitador-8588749977-8jfjm\" (UID: \"654a521c-2473-4a47-a492-5e4ca7aad6b5\") " pod="kuadrant-system/limitador-limitador-8588749977-8jfjm"
Sep 16 18:26:19.935344 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:19.935318    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ntf\" (UniqueName: \"kubernetes.io/projected/654a521c-2473-4a47-a492-5e4ca7aad6b5-kube-api-access-68ntf\") pod \"limitador-limitador-8588749977-8jfjm\" (UID: \"654a521c-2473-4a47-a492-5e4ca7aad6b5\") " pod="kuadrant-system/limitador-limitador-8588749977-8jfjm"
Sep 16 18:26:20.067787 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:20.067730    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-8jfjm"
Sep 16 18:26:20.168938 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:20.168904    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:26:20.172983 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:20.172963    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"
Sep 16 18:26:20.181062 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:20.181038    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:26:20.212992 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:20.212967    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:26:20.228905 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:20.228881    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b2c4f1de-925d-4090-9ed7-9532521ac154-config-file\") pod \"limitador-limitador-84994f7ffc-rvdrk\" (UID: \"b2c4f1de-925d-4090-9ed7-9532521ac154\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"
Sep 16 18:26:20.229016 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:20.228984    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wtkb\" (UniqueName: \"kubernetes.io/projected/b2c4f1de-925d-4090-9ed7-9532521ac154-kube-api-access-7wtkb\") pod \"limitador-limitador-84994f7ffc-rvdrk\" (UID: \"b2c4f1de-925d-4090-9ed7-9532521ac154\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"
Sep 16 18:26:20.329824 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:20.329769    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wtkb\" (UniqueName: \"kubernetes.io/projected/b2c4f1de-925d-4090-9ed7-9532521ac154-kube-api-access-7wtkb\") pod \"limitador-limitador-84994f7ffc-rvdrk\" (UID: \"b2c4f1de-925d-4090-9ed7-9532521ac154\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"
Sep 16 18:26:20.329824 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:20.329807    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b2c4f1de-925d-4090-9ed7-9532521ac154-config-file\") pod \"limitador-limitador-84994f7ffc-rvdrk\" (UID: \"b2c4f1de-925d-4090-9ed7-9532521ac154\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"
Sep 16 18:26:20.330351 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:20.330334    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b2c4f1de-925d-4090-9ed7-9532521ac154-config-file\") pod \"limitador-limitador-84994f7ffc-rvdrk\" (UID: \"b2c4f1de-925d-4090-9ed7-9532521ac154\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"
Sep 16 18:26:20.335899 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:20.335879    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8jfjm"]
Sep 16 18:26:20.338050 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:20.338026    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wtkb\" (UniqueName: \"kubernetes.io/projected/b2c4f1de-925d-4090-9ed7-9532521ac154-kube-api-access-7wtkb\") pod \"limitador-limitador-84994f7ffc-rvdrk\" (UID: \"b2c4f1de-925d-4090-9ed7-9532521ac154\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"
Sep 16 18:26:20.338347 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:26:20.338327    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654a521c_2473_4a47_a492_5e4ca7aad6b5.slice/crio-d19994a2da6c06d70f915144bc05cd8e277345e87c40ec7868c3c8d5f7b81042 WatchSource:0}: Error finding container d19994a2da6c06d70f915144bc05cd8e277345e87c40ec7868c3c8d5f7b81042: Status 404 returned error can't find the container with id d19994a2da6c06d70f915144bc05cd8e277345e87c40ec7868c3c8d5f7b81042
Sep 16 18:26:20.483749 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:20.483720    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"
Sep 16 18:26:20.506906 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:20.506875    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-8jfjm" event={"ID":"654a521c-2473-4a47-a492-5e4ca7aad6b5","Type":"ContainerStarted","Data":"d19994a2da6c06d70f915144bc05cd8e277345e87c40ec7868c3c8d5f7b81042"}
Sep 16 18:26:20.752597 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:20.752511    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:26:20.754601 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:26:20.754566    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c4f1de_925d_4090_9ed7_9532521ac154.slice/crio-fd006dbb2c9124e49c27fce395a189569ec65affe6ef2962090077490a8541fc WatchSource:0}: Error finding container fd006dbb2c9124e49c27fce395a189569ec65affe6ef2962090077490a8541fc: Status 404 returned error can't find the container with id fd006dbb2c9124e49c27fce395a189569ec65affe6ef2962090077490a8541fc
Sep 16 18:26:21.513466 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:21.513440    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-rvdrk" event={"ID":"b2c4f1de-925d-4090-9ed7-9532521ac154","Type":"ContainerStarted","Data":"fd006dbb2c9124e49c27fce395a189569ec65affe6ef2962090077490a8541fc"}
Sep 16 18:26:22.518931 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:22.518882    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-8jfjm" event={"ID":"654a521c-2473-4a47-a492-5e4ca7aad6b5","Type":"ContainerStarted","Data":"c469465d113a7cf1d0ebfb15662a8f1023be698fe88626404c2f06a43c198d95"}
Sep 16 18:26:22.519412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:22.519001    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-8jfjm"
Sep 16 18:26:22.520490 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:22.520452    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-rvdrk" event={"ID":"b2c4f1de-925d-4090-9ed7-9532521ac154","Type":"ContainerStarted","Data":"e8a181412359d495fe6bdfcf9d15970c8dd66146dd2190002cf4f4a98221715a"}
Sep 16 18:26:22.520617 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:22.520518    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"
Sep 16 18:26:22.536174 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:22.536124    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-8jfjm" podStartSLOduration=2.426286345 podStartE2EDuration="3.536111583s" totalImagesPullingTime="1.109825238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:26:22.535333999 +0000 UTC m=+606.094501833" watchObservedRunningTime="2026-09-16 18:26:22.536111583 +0000 UTC m=+606.095279416"
Sep 16 18:26:22.551272 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:22.551214    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-rvdrk" podStartSLOduration=1.856500145 podStartE2EDuration="2.551202501s" totalImagesPullingTime="694.702356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:26:22.549978361 +0000 UTC m=+606.109146391" watchObservedRunningTime="2026-09-16 18:26:22.551202501 +0000 UTC m=+606.110370365"
Sep 16 18:26:33.526723 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:33.526689    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-8jfjm"
Sep 16 18:26:33.527149 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:33.526760    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"
Sep 16 18:26:33.591764 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:33.591728    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8jfjm"]
Sep 16 18:26:33.591936 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:33.591899    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-8jfjm" podUID="654a521c-2473-4a47-a492-5e4ca7aad6b5" containerName="limitador" containerID="cri-o://c469465d113a7cf1d0ebfb15662a8f1023be698fe88626404c2f06a43c198d95" gracePeriod=30
Sep 16 18:26:34.139205 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.139179    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-8jfjm"
Sep 16 18:26:34.260670 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.260642    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/654a521c-2473-4a47-a492-5e4ca7aad6b5-config-file\") pod \"654a521c-2473-4a47-a492-5e4ca7aad6b5\" (UID: \"654a521c-2473-4a47-a492-5e4ca7aad6b5\") "
Sep 16 18:26:34.260823 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.260705    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68ntf\" (UniqueName: \"kubernetes.io/projected/654a521c-2473-4a47-a492-5e4ca7aad6b5-kube-api-access-68ntf\") pod \"654a521c-2473-4a47-a492-5e4ca7aad6b5\" (UID: \"654a521c-2473-4a47-a492-5e4ca7aad6b5\") "
Sep 16 18:26:34.261039 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.261018    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654a521c-2473-4a47-a492-5e4ca7aad6b5-config-file" (OuterVolumeSpecName: "config-file") pod "654a521c-2473-4a47-a492-5e4ca7aad6b5" (UID: "654a521c-2473-4a47-a492-5e4ca7aad6b5"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:26:34.262564 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.262542    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654a521c-2473-4a47-a492-5e4ca7aad6b5-kube-api-access-68ntf" (OuterVolumeSpecName: "kube-api-access-68ntf") pod "654a521c-2473-4a47-a492-5e4ca7aad6b5" (UID: "654a521c-2473-4a47-a492-5e4ca7aad6b5"). InnerVolumeSpecName "kube-api-access-68ntf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:26:34.362026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.361996    2554 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/654a521c-2473-4a47-a492-5e4ca7aad6b5-config-file\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:26:34.362026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.362019    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68ntf\" (UniqueName: \"kubernetes.io/projected/654a521c-2473-4a47-a492-5e4ca7aad6b5-kube-api-access-68ntf\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:26:34.577071 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.576988    2554 generic.go:356] "Generic (PLEG): container finished" podID="654a521c-2473-4a47-a492-5e4ca7aad6b5" containerID="c469465d113a7cf1d0ebfb15662a8f1023be698fe88626404c2f06a43c198d95" exitCode=0
Sep 16 18:26:34.577071 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.577037    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-8jfjm"
Sep 16 18:26:34.577502 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.577074    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-8jfjm" event={"ID":"654a521c-2473-4a47-a492-5e4ca7aad6b5","Type":"ContainerDied","Data":"c469465d113a7cf1d0ebfb15662a8f1023be698fe88626404c2f06a43c198d95"}
Sep 16 18:26:34.577502 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.577128    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-8jfjm" event={"ID":"654a521c-2473-4a47-a492-5e4ca7aad6b5","Type":"ContainerDied","Data":"d19994a2da6c06d70f915144bc05cd8e277345e87c40ec7868c3c8d5f7b81042"}
Sep 16 18:26:34.577502 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.577151    2554 scope.go:117] "RemoveContainer" containerID="c469465d113a7cf1d0ebfb15662a8f1023be698fe88626404c2f06a43c198d95"
Sep 16 18:26:34.587939 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.587889    2554 scope.go:117] "RemoveContainer" containerID="c469465d113a7cf1d0ebfb15662a8f1023be698fe88626404c2f06a43c198d95"
Sep 16 18:26:34.588364 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:26:34.588342    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c469465d113a7cf1d0ebfb15662a8f1023be698fe88626404c2f06a43c198d95\": container with ID starting with c469465d113a7cf1d0ebfb15662a8f1023be698fe88626404c2f06a43c198d95 not found: ID does not exist" containerID="c469465d113a7cf1d0ebfb15662a8f1023be698fe88626404c2f06a43c198d95"
Sep 16 18:26:34.588467 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.588369    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c469465d113a7cf1d0ebfb15662a8f1023be698fe88626404c2f06a43c198d95"} err="failed to get container status \"c469465d113a7cf1d0ebfb15662a8f1023be698fe88626404c2f06a43c198d95\": rpc error: code = NotFound desc = could not find container \"c469465d113a7cf1d0ebfb15662a8f1023be698fe88626404c2f06a43c198d95\": container with ID starting with c469465d113a7cf1d0ebfb15662a8f1023be698fe88626404c2f06a43c198d95 not found: ID does not exist"
Sep 16 18:26:34.597356 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.597336    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8jfjm"]
Sep 16 18:26:34.601326 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.601307    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-8jfjm"]
Sep 16 18:26:34.884185 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:34.884114    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="654a521c-2473-4a47-a492-5e4ca7aad6b5" path="/var/lib/kubelet/pods/654a521c-2473-4a47-a492-5e4ca7aad6b5/volumes"
Sep 16 18:26:37.261511 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.261476    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ln86k"]
Sep 16 18:26:37.262875 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.262852    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="654a521c-2473-4a47-a492-5e4ca7aad6b5" containerName="limitador"
Sep 16 18:26:37.263018 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.263007    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="654a521c-2473-4a47-a492-5e4ca7aad6b5" containerName="limitador"
Sep 16 18:26:37.263569 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.263295    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="654a521c-2473-4a47-a492-5e4ca7aad6b5" containerName="limitador"
Sep 16 18:26:37.269525 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.269498    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ln86k"
Sep 16 18:26:37.271934 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.271911    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ln86k"]
Sep 16 18:26:37.272279 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.272225    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-h9sxc\""
Sep 16 18:26:37.272391 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.272312    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 16 18:26:37.272391 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.272323    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:26:37.273175 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.273157    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 18:26:37.388693 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.388657    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn5lx\" (UniqueName: \"kubernetes.io/projected/d6dfc071-3732-4c49-986b-b121a18f5f52-kube-api-access-qn5lx\") pod \"postgres-69fb4594cf-ln86k\" (UID: \"d6dfc071-3732-4c49-986b-b121a18f5f52\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ln86k"
Sep 16 18:26:37.388841 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.388702    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6dfc071-3732-4c49-986b-b121a18f5f52-data\") pod \"postgres-69fb4594cf-ln86k\" (UID: \"d6dfc071-3732-4c49-986b-b121a18f5f52\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ln86k"
Sep 16 18:26:37.489447 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.489418    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn5lx\" (UniqueName: \"kubernetes.io/projected/d6dfc071-3732-4c49-986b-b121a18f5f52-kube-api-access-qn5lx\") pod \"postgres-69fb4594cf-ln86k\" (UID: \"d6dfc071-3732-4c49-986b-b121a18f5f52\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ln86k"
Sep 16 18:26:37.489570 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.489458    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6dfc071-3732-4c49-986b-b121a18f5f52-data\") pod \"postgres-69fb4594cf-ln86k\" (UID: \"d6dfc071-3732-4c49-986b-b121a18f5f52\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ln86k"
Sep 16 18:26:37.489800 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.489786    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6dfc071-3732-4c49-986b-b121a18f5f52-data\") pod \"postgres-69fb4594cf-ln86k\" (UID: \"d6dfc071-3732-4c49-986b-b121a18f5f52\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ln86k"
Sep 16 18:26:37.497153 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.497121    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn5lx\" (UniqueName: \"kubernetes.io/projected/d6dfc071-3732-4c49-986b-b121a18f5f52-kube-api-access-qn5lx\") pod \"postgres-69fb4594cf-ln86k\" (UID: \"d6dfc071-3732-4c49-986b-b121a18f5f52\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ln86k"
Sep 16 18:26:37.583432 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.583372    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ln86k"
Sep 16 18:26:37.853691 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:37.853666    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ln86k"]
Sep 16 18:26:37.856328 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:26:37.856290    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6dfc071_3732_4c49_986b_b121a18f5f52.slice/crio-9767957c99e6c18615d92a299bf9f7e6e404ae081073df49c9326d1e0624afed WatchSource:0}: Error finding container 9767957c99e6c18615d92a299bf9f7e6e404ae081073df49c9326d1e0624afed: Status 404 returned error can't find the container with id 9767957c99e6c18615d92a299bf9f7e6e404ae081073df49c9326d1e0624afed
Sep 16 18:26:38.602180 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:38.602126    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ln86k" event={"ID":"d6dfc071-3732-4c49-986b-b121a18f5f52","Type":"ContainerStarted","Data":"9767957c99e6c18615d92a299bf9f7e6e404ae081073df49c9326d1e0624afed"}
Sep 16 18:26:42.623857 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:42.623820    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ln86k" event={"ID":"d6dfc071-3732-4c49-986b-b121a18f5f52","Type":"ContainerStarted","Data":"93b22ec7c666c41dca86b3a71db452623d370803b34c81b61933e6c4c70d645e"}
Sep 16 18:26:42.624290 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:42.623888    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ln86k"
Sep 16 18:26:42.639316 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:42.639207    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ln86k" podStartSLOduration=1.112874016 podStartE2EDuration="5.639190441s" totalImagesPullingTime="4.526316425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:26:42.638170585 +0000 UTC m=+626.197338420" watchObservedRunningTime="2026-09-16 18:26:42.639190441 +0000 UTC m=+626.198358275"
Sep 16 18:26:48.656056 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:48.656021    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ln86k"
Sep 16 18:26:49.518022 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:49.517986    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-89cfcb966-c22jb"]
Sep 16 18:26:49.524061 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:49.524041    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-89cfcb966-c22jb"
Sep 16 18:26:49.526503 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:49.526480    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 18:26:49.526606 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:49.526560    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vh4db\""
Sep 16 18:26:49.530980 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:49.530602    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-89cfcb966-c22jb"]
Sep 16 18:26:49.701185 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:49.701150    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06c0a169-e22c-4973-a5a4-c46891d43942-tls-cert\") pod \"authorino-89cfcb966-c22jb\" (UID: \"06c0a169-e22c-4973-a5a4-c46891d43942\") " pod="kuadrant-system/authorino-89cfcb966-c22jb"
Sep 16 18:26:49.701575 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:49.701297    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq2jm\" (UniqueName: \"kubernetes.io/projected/06c0a169-e22c-4973-a5a4-c46891d43942-kube-api-access-nq2jm\") pod \"authorino-89cfcb966-c22jb\" (UID: \"06c0a169-e22c-4973-a5a4-c46891d43942\") " pod="kuadrant-system/authorino-89cfcb966-c22jb"
Sep 16 18:26:49.801886 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:49.801806    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq2jm\" (UniqueName: \"kubernetes.io/projected/06c0a169-e22c-4973-a5a4-c46891d43942-kube-api-access-nq2jm\") pod \"authorino-89cfcb966-c22jb\" (UID: \"06c0a169-e22c-4973-a5a4-c46891d43942\") " pod="kuadrant-system/authorino-89cfcb966-c22jb"
Sep 16 18:26:49.801886 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:49.801849    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06c0a169-e22c-4973-a5a4-c46891d43942-tls-cert\") pod \"authorino-89cfcb966-c22jb\" (UID: \"06c0a169-e22c-4973-a5a4-c46891d43942\") " pod="kuadrant-system/authorino-89cfcb966-c22jb"
Sep 16 18:26:49.804193 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:49.804172    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06c0a169-e22c-4973-a5a4-c46891d43942-tls-cert\") pod \"authorino-89cfcb966-c22jb\" (UID: \"06c0a169-e22c-4973-a5a4-c46891d43942\") " pod="kuadrant-system/authorino-89cfcb966-c22jb"
Sep 16 18:26:49.809668 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:49.809649    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq2jm\" (UniqueName: \"kubernetes.io/projected/06c0a169-e22c-4973-a5a4-c46891d43942-kube-api-access-nq2jm\") pod \"authorino-89cfcb966-c22jb\" (UID: \"06c0a169-e22c-4973-a5a4-c46891d43942\") " pod="kuadrant-system/authorino-89cfcb966-c22jb"
Sep 16 18:26:49.832960 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:49.832936    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-89cfcb966-c22jb"
Sep 16 18:26:50.106362 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:50.106338    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-89cfcb966-c22jb"]
Sep 16 18:26:50.108949 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:26:50.108911    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06c0a169_e22c_4973_a5a4_c46891d43942.slice/crio-d54fdfb8bab7e1759126cba0a81d7a157b224b8670da23b7266250bd35d2ce08 WatchSource:0}: Error finding container d54fdfb8bab7e1759126cba0a81d7a157b224b8670da23b7266250bd35d2ce08: Status 404 returned error can't find the container with id d54fdfb8bab7e1759126cba0a81d7a157b224b8670da23b7266250bd35d2ce08
Sep 16 18:26:50.665282 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:50.665224    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-89cfcb966-c22jb" event={"ID":"06c0a169-e22c-4973-a5a4-c46891d43942","Type":"ContainerStarted","Data":"d54fdfb8bab7e1759126cba0a81d7a157b224b8670da23b7266250bd35d2ce08"}
Sep 16 18:26:54.687383 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:54.687342    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-89cfcb966-c22jb" event={"ID":"06c0a169-e22c-4973-a5a4-c46891d43942","Type":"ContainerStarted","Data":"82a73bf6e5db45ae36aba46b7f437304447734f85c7466789cd8bbb2bd525ac0"}
Sep 16 18:26:54.704051 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:26:54.703990    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-89cfcb966-c22jb" podStartSLOduration=1.6449047380000001 podStartE2EDuration="5.703971898s" totalImagesPullingTime="4.05906716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:26:54.70388176 +0000 UTC m=+638.263049598" watchObservedRunningTime="2026-09-16 18:26:54.703971898 +0000 UTC m=+638.263139735"
Sep 16 18:27:37.605115 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:27:37.605079    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:27:55.405007 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:27:55.404970    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:28:02.108024 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:02.107978    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:28:25.302787 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.302710    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"]
Sep 16 18:28:25.306067 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.306049    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.308621 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.308591    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 18:28:25.308761 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.308655    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9ct5h\""
Sep 16 18:28:25.309673 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.309651    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 16 18:28:25.309779 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.309694    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 18:28:25.317946 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.317920    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"]
Sep 16 18:28:25.439507 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.439469    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.439507 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.439507    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.439728 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.439525    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.439728 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.439610    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8922g\" (UniqueName: \"kubernetes.io/projected/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-kube-api-access-8922g\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.439728 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.439661    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.439728 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.439716    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.540628 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.540590    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.540838 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.540653    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.540838 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.540698    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.540838 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.540717    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.540838 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.540737    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.540838 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.540787    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8922g\" (UniqueName: \"kubernetes.io/projected/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-kube-api-access-8922g\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.541162 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.541134    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.541408 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.541383    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.541501 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.541436    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.543117 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.543090    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.543370 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.543351    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.548579 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.548555    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8922g\" (UniqueName: \"kubernetes.io/projected/dffdb8a6-8524-4413-9fc2-5e4cffa61c80-kube-api-access-8922g\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb\" (UID: \"dffdb8a6-8524-4413-9fc2-5e4cffa61c80\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.616661 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.616572    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:25.892101 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.892020    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"]
Sep 16 18:28:25.895728 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:28:25.895696    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffdb8a6_8524_4413_9fc2_5e4cffa61c80.slice/crio-1c6bfb0469e39991f31c77ba069343f462645eb60e7fa572d6bfa9e203b1e650 WatchSource:0}: Error finding container 1c6bfb0469e39991f31c77ba069343f462645eb60e7fa572d6bfa9e203b1e650: Status 404 returned error can't find the container with id 1c6bfb0469e39991f31c77ba069343f462645eb60e7fa572d6bfa9e203b1e650
Sep 16 18:28:25.897106 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:25.897087    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:28:26.119718 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:26.119647    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb" event={"ID":"dffdb8a6-8524-4413-9fc2-5e4cffa61c80","Type":"ContainerStarted","Data":"1c6bfb0469e39991f31c77ba069343f462645eb60e7fa572d6bfa9e203b1e650"}
Sep 16 18:28:26.507978 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:26.507225    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:28:27.125702 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:27.125661    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb" event={"ID":"dffdb8a6-8524-4413-9fc2-5e4cffa61c80","Type":"ContainerStarted","Data":"241eb4a68713f3f014e6b5fd46c980e89598ca64481730662c90a7856a41c1e3"}
Sep 16 18:28:27.147423 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:27.147373    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb" podStartSLOduration=1.013991286 podStartE2EDuration="2.147354717s" totalImagesPullingTime="1.133363431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:27.145086896 +0000 UTC m=+730.704254734" watchObservedRunningTime="2026-09-16 18:28:27.147354717 +0000 UTC m=+730.706522550"
Sep 16 18:28:31.599284 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:31.599243    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:28:35.617064 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:35.617006    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:35.617565 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:35.617095    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:35.629503 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:35.629476    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:36.180992 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:36.180963    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb"
Sep 16 18:28:37.624874 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:37.624831    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:28:57.131785 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.131752    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"]
Sep 16 18:28:57.136626 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.136593    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.146391 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.146373    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 16 18:28:57.170303 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.170276    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"]
Sep 16 18:28:57.199817 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.199785    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7j6z\" (UniqueName: \"kubernetes.io/projected/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-kube-api-access-v7j6z\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.199958 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.199836    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.199958 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.199862    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.199958 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.199914    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.199958 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.199953    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.200112 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.199995    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.300890 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.300833    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.300890 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.300892    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.301130 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.300940    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.301130 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.300978    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.301130 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.301035    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.301130 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.301067    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7j6z\" (UniqueName: \"kubernetes.io/projected/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-kube-api-access-v7j6z\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.301364 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.301276    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.301364 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.301321    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.301457 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.301401    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.303222 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.303196    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.303500 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.303482    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.311527 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.311506    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7j6z\" (UniqueName: \"kubernetes.io/projected/2fd5a91b-c7c4-4fe0-b86e-54210860d8a2-kube-api-access-v7j6z\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-98mxv\" (UID: \"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.449469 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.449384    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:28:57.726068 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:57.725978    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"]
Sep 16 18:28:57.728998 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:28:57.728973    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd5a91b_c7c4_4fe0_b86e_54210860d8a2.slice/crio-f517d1f9fb7a42ee5bea094ac200f1c5fc5b5ae68e6fa56701291f27d23f677c WatchSource:0}: Error finding container f517d1f9fb7a42ee5bea094ac200f1c5fc5b5ae68e6fa56701291f27d23f677c: Status 404 returned error can't find the container with id f517d1f9fb7a42ee5bea094ac200f1c5fc5b5ae68e6fa56701291f27d23f677c
Sep 16 18:28:58.279513 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:58.279476    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv" event={"ID":"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2","Type":"ContainerStarted","Data":"66004677b1dd337c0bfe28726d07e1245a860f245944e79708623f6d8dc5b4b5"}
Sep 16 18:28:58.279901 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:58.279538    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv" event={"ID":"2fd5a91b-c7c4-4fe0-b86e-54210860d8a2","Type":"ContainerStarted","Data":"f517d1f9fb7a42ee5bea094ac200f1c5fc5b5ae68e6fa56701291f27d23f677c"}
Sep 16 18:28:58.306883 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:58.306838    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv" podStartSLOduration=1.1258461610000001 podStartE2EDuration="1.306822938s" totalImagesPullingTime="180.976777ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:58.302781574 +0000 UTC m=+761.861949409" watchObservedRunningTime="2026-09-16 18:28:58.306822938 +0000 UTC m=+761.865990805"
Sep 16 18:28:59.105145 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:28:59.105108    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:29:07.450216 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:29:07.450167    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:29:07.450742 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:29:07.450275    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:29:07.462710 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:29:07.462685    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:29:08.339036 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:29:08.339005    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-98mxv"
Sep 16 18:29:19.510994 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:29:19.510955    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:30:00.152551 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:00.152513    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826390-rsnjk"]
Sep 16 18:30:00.155933 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:00.155916    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826390-rsnjk"
Sep 16 18:30:00.158619 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:00.158599    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lfw6p\""
Sep 16 18:30:00.164825 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:00.164797    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826390-rsnjk"]
Sep 16 18:30:00.166312 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:00.166293    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826390-rsnjk"
Sep 16 18:30:00.442057 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:00.442027    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826390-rsnjk"]
Sep 16 18:30:00.444584 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:30:00.444552    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fdbf2a_51fc_4498_abc1_92e00c38ebc9.slice/crio-50a62194eafce48afa2e9343719c8d928c6ea9570584c1449f11722984914dc9 WatchSource:0}: Error finding container 50a62194eafce48afa2e9343719c8d928c6ea9570584c1449f11722984914dc9: Status 404 returned error can't find the container with id 50a62194eafce48afa2e9343719c8d928c6ea9570584c1449f11722984914dc9
Sep 16 18:30:00.579254 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:00.579201    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826390-rsnjk" event={"ID":"15fdbf2a-51fc-4498-abc1-92e00c38ebc9","Type":"ContainerStarted","Data":"50a62194eafce48afa2e9343719c8d928c6ea9570584c1449f11722984914dc9"}
Sep 16 18:30:03.597810 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:03.597771    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826390-rsnjk" event={"ID":"15fdbf2a-51fc-4498-abc1-92e00c38ebc9","Type":"ContainerStarted","Data":"af6b8f8bbb042eb4e048dd2686be338a9a93437c8cde5b40404d839214e2d230"}
Sep 16 18:30:03.616393 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:03.616341    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826390-rsnjk" podStartSLOduration=1.2457775309999999 podStartE2EDuration="3.616325914s" totalImagesPullingTime="2.370548383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:30:03.615192576 +0000 UTC m=+827.174360409" watchObservedRunningTime="2026-09-16 18:30:03.616325914 +0000 UTC m=+827.175493747"
Sep 16 18:30:04.604528 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:04.604492    2554 generic.go:356] "Generic (PLEG): container finished" podID="15fdbf2a-51fc-4498-abc1-92e00c38ebc9" containerID="af6b8f8bbb042eb4e048dd2686be338a9a93437c8cde5b40404d839214e2d230" exitCode=0
Sep 16 18:30:04.604922 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:04.604580    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826390-rsnjk" event={"ID":"15fdbf2a-51fc-4498-abc1-92e00c38ebc9","Type":"ContainerDied","Data":"af6b8f8bbb042eb4e048dd2686be338a9a93437c8cde5b40404d839214e2d230"}
Sep 16 18:30:05.742148 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:05.742124    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826390-rsnjk"
Sep 16 18:30:06.615822 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:06.615792    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826390-rsnjk"
Sep 16 18:30:06.616015 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:06.615834    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826390-rsnjk" event={"ID":"15fdbf2a-51fc-4498-abc1-92e00c38ebc9","Type":"ContainerDied","Data":"50a62194eafce48afa2e9343719c8d928c6ea9570584c1449f11722984914dc9"}
Sep 16 18:30:06.616015 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:06.615890    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50a62194eafce48afa2e9343719c8d928c6ea9570584c1449f11722984914dc9"
Sep 16 18:30:07.868654 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:07.868607    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-89cfcb966-c22jb"]
Sep 16 18:30:07.869087 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:07.868894    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-89cfcb966-c22jb" podUID="06c0a169-e22c-4973-a5a4-c46891d43942" containerName="authorino" containerID="cri-o://82a73bf6e5db45ae36aba46b7f437304447734f85c7466789cd8bbb2bd525ac0" gracePeriod=30
Sep 16 18:30:08.126741 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.126690    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-89cfcb966-c22jb"
Sep 16 18:30:08.236259 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.236200    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nq2jm\" (UniqueName: \"kubernetes.io/projected/06c0a169-e22c-4973-a5a4-c46891d43942-kube-api-access-nq2jm\") pod \"06c0a169-e22c-4973-a5a4-c46891d43942\" (UID: \"06c0a169-e22c-4973-a5a4-c46891d43942\") "
Sep 16 18:30:08.236440 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.236323    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06c0a169-e22c-4973-a5a4-c46891d43942-tls-cert\") pod \"06c0a169-e22c-4973-a5a4-c46891d43942\" (UID: \"06c0a169-e22c-4973-a5a4-c46891d43942\") "
Sep 16 18:30:08.238437 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.238399    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c0a169-e22c-4973-a5a4-c46891d43942-kube-api-access-nq2jm" (OuterVolumeSpecName: "kube-api-access-nq2jm") pod "06c0a169-e22c-4973-a5a4-c46891d43942" (UID: "06c0a169-e22c-4973-a5a4-c46891d43942"). InnerVolumeSpecName "kube-api-access-nq2jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:30:08.247091 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.247057    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c0a169-e22c-4973-a5a4-c46891d43942-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "06c0a169-e22c-4973-a5a4-c46891d43942" (UID: "06c0a169-e22c-4973-a5a4-c46891d43942"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:30:08.337226 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.337191    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq2jm\" (UniqueName: \"kubernetes.io/projected/06c0a169-e22c-4973-a5a4-c46891d43942-kube-api-access-nq2jm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:30:08.337226 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.337221    2554 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06c0a169-e22c-4973-a5a4-c46891d43942-tls-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:30:08.628075 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.628043    2554 generic.go:356] "Generic (PLEG): container finished" podID="06c0a169-e22c-4973-a5a4-c46891d43942" containerID="82a73bf6e5db45ae36aba46b7f437304447734f85c7466789cd8bbb2bd525ac0" exitCode=0
Sep 16 18:30:08.628226 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.628090    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-89cfcb966-c22jb"
Sep 16 18:30:08.628226 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.628131    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-89cfcb966-c22jb" event={"ID":"06c0a169-e22c-4973-a5a4-c46891d43942","Type":"ContainerDied","Data":"82a73bf6e5db45ae36aba46b7f437304447734f85c7466789cd8bbb2bd525ac0"}
Sep 16 18:30:08.628226 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.628178    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-89cfcb966-c22jb" event={"ID":"06c0a169-e22c-4973-a5a4-c46891d43942","Type":"ContainerDied","Data":"d54fdfb8bab7e1759126cba0a81d7a157b224b8670da23b7266250bd35d2ce08"}
Sep 16 18:30:08.628226 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.628202    2554 scope.go:117] "RemoveContainer" containerID="82a73bf6e5db45ae36aba46b7f437304447734f85c7466789cd8bbb2bd525ac0"
Sep 16 18:30:08.637225 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.637207    2554 scope.go:117] "RemoveContainer" containerID="82a73bf6e5db45ae36aba46b7f437304447734f85c7466789cd8bbb2bd525ac0"
Sep 16 18:30:08.637523 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:08.637498    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a73bf6e5db45ae36aba46b7f437304447734f85c7466789cd8bbb2bd525ac0\": container with ID starting with 82a73bf6e5db45ae36aba46b7f437304447734f85c7466789cd8bbb2bd525ac0 not found: ID does not exist" containerID="82a73bf6e5db45ae36aba46b7f437304447734f85c7466789cd8bbb2bd525ac0"
Sep 16 18:30:08.637619 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.637539    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a73bf6e5db45ae36aba46b7f437304447734f85c7466789cd8bbb2bd525ac0"} err="failed to get container status \"82a73bf6e5db45ae36aba46b7f437304447734f85c7466789cd8bbb2bd525ac0\": rpc error: code = NotFound desc = could not find container \"82a73bf6e5db45ae36aba46b7f437304447734f85c7466789cd8bbb2bd525ac0\": container with ID starting with 82a73bf6e5db45ae36aba46b7f437304447734f85c7466789cd8bbb2bd525ac0 not found: ID does not exist"
Sep 16 18:30:08.648310 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.648283    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-89cfcb966-c22jb"]
Sep 16 18:30:08.651724 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.651700    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-89cfcb966-c22jb"]
Sep 16 18:30:08.883275 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:08.883185    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06c0a169-e22c-4973-a5a4-c46891d43942" path="/var/lib/kubelet/pods/06c0a169-e22c-4973-a5a4-c46891d43942/volumes"
Sep 16 18:30:29.060908 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.060873    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"]
Sep 16 18:30:29.061412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.061279    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15fdbf2a-51fc-4498-abc1-92e00c38ebc9" containerName="cleanup"
Sep 16 18:30:29.061412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.061290    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="15fdbf2a-51fc-4498-abc1-92e00c38ebc9" containerName="cleanup"
Sep 16 18:30:29.061412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.061305    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06c0a169-e22c-4973-a5a4-c46891d43942" containerName="authorino"
Sep 16 18:30:29.061412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.061312    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="06c0a169-e22c-4973-a5a4-c46891d43942" containerName="authorino"
Sep 16 18:30:29.061412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.061370    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="15fdbf2a-51fc-4498-abc1-92e00c38ebc9" containerName="cleanup"
Sep 16 18:30:29.061412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.061378    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="06c0a169-e22c-4973-a5a4-c46891d43942" containerName="authorino"
Sep 16 18:30:29.065549 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.065529    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.069162 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.069137    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:30:29.069318 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.069173    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:30:29.069994 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.069972    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:30:29.070089 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.070071    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a24d09-openshift-default-dockercfg-l2pjq\""
Sep 16 18:30:29.089040 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.089004    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"]
Sep 16 18:30:29.121480 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.121439    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-workload-socket\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.121656 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.121494    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-envoy\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.121656 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.121521    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istiod-ca-cert\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.121656 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.121575    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-token\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.121656 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.121621    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-data\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.121832 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.121657    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dmkv\" (UniqueName: \"kubernetes.io/projected/335a6bc4-c505-4fab-b12f-d2ccc04e971d-kube-api-access-9dmkv\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.121832 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.121718    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-workload-certs\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.121832 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.121784    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-credential-socket\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.121959 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.121844    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-podinfo\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.222664 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.222623    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-credential-socket\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.222664 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.222660    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-podinfo\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.222871 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.222694    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-workload-socket\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.222871 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.222711    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-envoy\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.222871 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.222727    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istiod-ca-cert\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.222871 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.222753    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-token\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.222871 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.222795    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-data\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.223467 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.222963    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dmkv\" (UniqueName: \"kubernetes.io/projected/335a6bc4-c505-4fab-b12f-d2ccc04e971d-kube-api-access-9dmkv\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.223467 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.223021    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-workload-certs\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.223467 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.223191    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-data\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.223467 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.223377    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-workload-socket\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.223467 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.223420    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-credential-socket\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.223706 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.223567    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istiod-ca-cert\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.223764 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.223702    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-workload-certs\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.225317 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.225289    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-envoy\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.225938 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.225912    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-podinfo\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.244163 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.244130    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dmkv\" (UniqueName: \"kubernetes.io/projected/335a6bc4-c505-4fab-b12f-d2ccc04e971d-kube-api-access-9dmkv\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.245280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.245257    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-token\") pod \"e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") " pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.375328 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.375221    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:29.658820 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.658793    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"]
Sep 16 18:30:29.660964 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:30:29.660936    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335a6bc4_c505_4fab_b12f_d2ccc04e971d.slice/crio-c91a610431414ef2d4962192de2f66fec5c9218f5f3aee0e49bb5cfefe892f02 WatchSource:0}: Error finding container c91a610431414ef2d4962192de2f66fec5c9218f5f3aee0e49bb5cfefe892f02: Status 404 returned error can't find the container with id c91a610431414ef2d4962192de2f66fec5c9218f5f3aee0e49bb5cfefe892f02
Sep 16 18:30:29.733475 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:29.733436    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm" event={"ID":"335a6bc4-c505-4fab-b12f-d2ccc04e971d","Type":"ContainerStarted","Data":"c91a610431414ef2d4962192de2f66fec5c9218f5f3aee0e49bb5cfefe892f02"}
Sep 16 18:30:30.621388 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:30.621350    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7718-4f4227"]
Sep 16 18:30:30.626054 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:30.626032    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7718-4f4227"
Sep 16 18:30:30.628443 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:30.628418    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xhq6g\""
Sep 16 18:30:30.631711 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:30.631683    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7718-4f4227"]
Sep 16 18:30:30.635822 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:30.635800    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fn6\" (UniqueName: \"kubernetes.io/projected/be7549cf-50d4-4440-9154-76a39dd09655-kube-api-access-s7fn6\") pod \"test-curl-7718-4f4227\" (UID: \"be7549cf-50d4-4440-9154-76a39dd09655\") " pod="opendatahub/test-curl-7718-4f4227"
Sep 16 18:30:30.736923 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:30.736892    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fn6\" (UniqueName: \"kubernetes.io/projected/be7549cf-50d4-4440-9154-76a39dd09655-kube-api-access-s7fn6\") pod \"test-curl-7718-4f4227\" (UID: \"be7549cf-50d4-4440-9154-76a39dd09655\") " pod="opendatahub/test-curl-7718-4f4227"
Sep 16 18:30:30.747165 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:30.747136    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fn6\" (UniqueName: \"kubernetes.io/projected/be7549cf-50d4-4440-9154-76a39dd09655-kube-api-access-s7fn6\") pod \"test-curl-7718-4f4227\" (UID: \"be7549cf-50d4-4440-9154-76a39dd09655\") " pod="opendatahub/test-curl-7718-4f4227"
Sep 16 18:30:30.937946 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:30.937717    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7718-4f4227"
Sep 16 18:30:31.227509 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:31.227477    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7718-4f4227"]
Sep 16 18:30:31.229873 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:30:31.229824    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe7549cf_50d4_4440_9154_76a39dd09655.slice/crio-d0ef147f43dda767ce3355793cbf268be2d8c428e12f507a9a9884f9efb0d57b WatchSource:0}: Error finding container d0ef147f43dda767ce3355793cbf268be2d8c428e12f507a9a9884f9efb0d57b: Status 404 returned error can't find the container with id d0ef147f43dda767ce3355793cbf268be2d8c428e12f507a9a9884f9efb0d57b
Sep 16 18:30:31.754372 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:31.754330    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7718-4f4227" event={"ID":"be7549cf-50d4-4440-9154-76a39dd09655","Type":"ContainerStarted","Data":"d0ef147f43dda767ce3355793cbf268be2d8c428e12f507a9a9884f9efb0d57b"}
Sep 16 18:30:31.819265 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:31.819215    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:30:32.636448 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:32.636410    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:30:32.636545 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:32.636489    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:30:32.636545 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:32.636521    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:30:32.761797 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:32.761752    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm" event={"ID":"335a6bc4-c505-4fab-b12f-d2ccc04e971d","Type":"ContainerStarted","Data":"6f267a416e3af69e03dbdc905a8b756371124c27fe044f0c01a68b5410bb2668"}
Sep 16 18:30:32.763479 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:32.763449    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7718-4f4227" event={"ID":"be7549cf-50d4-4440-9154-76a39dd09655","Type":"ContainerStarted","Data":"409f7db8938799b2f0f1acfe24628f10fe19eeb5e1cc2fdb51b543436de9a57c"}
Sep 16 18:30:32.785414 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:32.785355    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm" podStartSLOduration=0.812071458 podStartE2EDuration="3.785335693s" totalImagesPullingTime="2.973264235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:30:32.782465758 +0000 UTC m=+856.341633592" watchObservedRunningTime="2026-09-16 18:30:32.785335693 +0000 UTC m=+856.344503524"
Sep 16 18:30:32.798380 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:32.798330    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7718-4f4227" podStartSLOduration=1.396606882 podStartE2EDuration="2.798313708s" totalImagesPullingTime="1.401706826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:30:32.796284185 +0000 UTC m=+856.355452017" watchObservedRunningTime="2026-09-16 18:30:32.798313708 +0000 UTC m=+856.357481540"
Sep 16 18:30:33.376304 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:33.376228    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:33.381374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:33.381344    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:33.478326 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:33.478295    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7718-4f4227"]
Sep 16 18:30:33.482319 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:33.482287    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7718-4f4227"]
Sep 16 18:30:33.606993 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:33.606904    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7718-041ca3"]
Sep 16 18:30:33.607309 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:33.607294    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be7549cf-50d4-4440-9154-76a39dd09655" containerName="test-curl-7718-4f4227"
Sep 16 18:30:33.607364 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:33.607310    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="be7549cf-50d4-4440-9154-76a39dd09655" containerName="test-curl-7718-4f4227"
Sep 16 18:30:33.607400 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:33.607392    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="be7549cf-50d4-4440-9154-76a39dd09655" containerName="test-curl-7718-4f4227"
Sep 16 18:30:33.610614 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:33.610596    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7718-041ca3"
Sep 16 18:30:33.614608 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:33.614579    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7718-041ca3"]
Sep 16 18:30:33.667109 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:33.667075    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvh9r\" (UniqueName: \"kubernetes.io/projected/674a6211-f6cd-44da-b6af-053dbcb9f052-kube-api-access-pvh9r\") pod \"test-curl-7718-041ca3\" (UID: \"674a6211-f6cd-44da-b6af-053dbcb9f052\") " pod="opendatahub/test-curl-7718-041ca3"
Sep 16 18:30:33.768150 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:33.768113    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvh9r\" (UniqueName: \"kubernetes.io/projected/674a6211-f6cd-44da-b6af-053dbcb9f052-kube-api-access-pvh9r\") pod \"test-curl-7718-041ca3\" (UID: \"674a6211-f6cd-44da-b6af-053dbcb9f052\") " pod="opendatahub/test-curl-7718-041ca3"
Sep 16 18:30:33.769004 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:33.768982    2554 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7718-4f4227" container="test-curl-7718-4f4227"
Sep 16 18:30:33.769164 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:33.769142    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:33.769990 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:33.769972    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:30:33.776461 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:33.776440    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvh9r\" (UniqueName: \"kubernetes.io/projected/674a6211-f6cd-44da-b6af-053dbcb9f052-kube-api-access-pvh9r\") pod \"test-curl-7718-041ca3\" (UID: \"674a6211-f6cd-44da-b6af-053dbcb9f052\") " pod="opendatahub/test-curl-7718-041ca3"
Sep 16 18:30:33.921772 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:33.921684    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7718-041ca3"
Sep 16 18:30:34.195240 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:34.195176    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7718-041ca3"]
Sep 16 18:30:34.197784 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:30:34.197755    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod674a6211_f6cd_44da_b6af_053dbcb9f052.slice/crio-8932bfabfb914b11a4ede9b8fcb421b072c4d119de43ea9c25061e235ffbb0ca WatchSource:0}: Error finding container 8932bfabfb914b11a4ede9b8fcb421b072c4d119de43ea9c25061e235ffbb0ca: Status 404 returned error can't find the container with id 8932bfabfb914b11a4ede9b8fcb421b072c4d119de43ea9c25061e235ffbb0ca
Sep 16 18:30:34.776668 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:34.776625    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7718-041ca3" event={"ID":"674a6211-f6cd-44da-b6af-053dbcb9f052","Type":"ContainerStarted","Data":"9ea34b6c3910ec0cc7b76bb7779f6bb9c2925160caaf615aff3bea253bb8b4c7"}
Sep 16 18:30:34.776668 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:34.776684    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7718-041ca3" event={"ID":"674a6211-f6cd-44da-b6af-053dbcb9f052","Type":"ContainerStarted","Data":"8932bfabfb914b11a4ede9b8fcb421b072c4d119de43ea9c25061e235ffbb0ca"}
Sep 16 18:30:34.777183 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:34.776838    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7718-4f4227" podUID="be7549cf-50d4-4440-9154-76a39dd09655" containerName="test-curl-7718-4f4227" containerID="cri-o://409f7db8938799b2f0f1acfe24628f10fe19eeb5e1cc2fdb51b543436de9a57c" gracePeriod=2
Sep 16 18:30:34.791055 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:34.791004    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7718-041ca3" podStartSLOduration=1.790990384 podStartE2EDuration="1.790990384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:30:34.789920012 +0000 UTC m=+858.349087843" watchObservedRunningTime="2026-09-16 18:30:34.790990384 +0000 UTC m=+858.350158213"
Sep 16 18:30:34.791679 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:34.791653    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7718-4f4227\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" podUID="be7549cf-50d4-4440-9154-76a39dd09655" pod="opendatahub/test-curl-7718-4f4227"
Sep 16 18:30:35.490936 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:35.490896    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7718-041ca3"]
Sep 16 18:30:35.498045 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:35.498011    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7718-041ca3"]
Sep 16 18:30:35.782002 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:35.781958    2554 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7718-041ca3" container="test-curl-7718-041ca3"
Sep 16 18:30:36.064973 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.064887    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"]
Sep 16 18:30:36.065315 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.065299    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="674a6211-f6cd-44da-b6af-053dbcb9f052" containerName="test-curl-7718-041ca3"
Sep 16 18:30:36.065315 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.065314    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="674a6211-f6cd-44da-b6af-053dbcb9f052" containerName="test-curl-7718-041ca3"
Sep 16 18:30:36.065434 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.065394    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="674a6211-f6cd-44da-b6af-053dbcb9f052" containerName="test-curl-7718-041ca3"
Sep 16 18:30:36.071996 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.071974    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:30:36.074799 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.074775    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-6942a4-serving-cert\""
Sep 16 18:30:36.074894 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.074853    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7l879\""
Sep 16 18:30:36.075526 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.075504    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 18:30:36.080100 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.080055    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"]
Sep 16 18:30:36.189355 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.189320    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8b9a6dd-c92f-4782-973b-88d41b05bac7-metrics-certs\") pod \"maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c\" (UID: \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:30:36.189355 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.189361    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b9a6dd-c92f-4782-973b-88d41b05bac7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c\" (UID: \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:30:36.189578 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.189464    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwwrv\" (UniqueName: \"kubernetes.io/projected/e8b9a6dd-c92f-4782-973b-88d41b05bac7-kube-api-access-nwwrv\") pod \"maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c\" (UID: \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:30:36.189578 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.189509    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8b9a6dd-c92f-4782-973b-88d41b05bac7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c\" (UID: \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:30:36.290540 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.290506    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwwrv\" (UniqueName: \"kubernetes.io/projected/e8b9a6dd-c92f-4782-973b-88d41b05bac7-kube-api-access-nwwrv\") pod \"maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c\" (UID: \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:30:36.290724 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.290546    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8b9a6dd-c92f-4782-973b-88d41b05bac7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c\" (UID: \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:30:36.290724 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.290617    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8b9a6dd-c92f-4782-973b-88d41b05bac7-metrics-certs\") pod \"maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c\" (UID: \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:30:36.290724 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.290653    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b9a6dd-c92f-4782-973b-88d41b05bac7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c\" (UID: \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:30:36.291550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.291526    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b9a6dd-c92f-4782-973b-88d41b05bac7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c\" (UID: \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:30:36.293622 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.293597    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8b9a6dd-c92f-4782-973b-88d41b05bac7-metrics-certs\") pod \"maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c\" (UID: \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:30:36.293721 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.293703    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8b9a6dd-c92f-4782-973b-88d41b05bac7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c\" (UID: \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:30:36.314257 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.314218    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwwrv\" (UniqueName: \"kubernetes.io/projected/e8b9a6dd-c92f-4782-973b-88d41b05bac7-kube-api-access-nwwrv\") pod \"maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c\" (UID: \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:30:36.336530 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.336461    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"]
Sep 16 18:30:36.340440 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.340413    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.344670 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:36.344635    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" podUID="627dd71a-bccc-4627-b52f-6914d4060a43" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.344780 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:36.344686    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-autores-305998-openshift-default-dockercfg-bxkzs\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"e2e-autores-305998-openshift-default-dockercfg-bxkzs\"" type="*v1.Secret"
Sep 16 18:30:36.363194 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.363166    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"]
Sep 16 18:30:36.384463 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.384436    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:30:36.391757 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.391732    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-istio-envoy\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.391870 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.391779    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627dd71a-bccc-4627-b52f-6914d4060a43-istiod-ca-cert\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.391870 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.391823    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627dd71a-bccc-4627-b52f-6914d4060a43-istio-podinfo\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.391870 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.391848    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-credential-socket\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.391870 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.391864    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7mpn\" (UniqueName: \"kubernetes.io/projected/627dd71a-bccc-4627-b52f-6914d4060a43-kube-api-access-s7mpn\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.392073 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.391975    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-workload-certs\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.392073 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.392017    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-workload-socket\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.392073 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.392041    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627dd71a-bccc-4627-b52f-6914d4060a43-istio-token\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.392214 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.392074    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-istio-data\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.492619 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.492583    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-workload-socket\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.492619 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.492620    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627dd71a-bccc-4627-b52f-6914d4060a43-istio-token\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.492839 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.492677    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-istio-data\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.492839 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.492728    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-istio-envoy\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.492839 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.492766    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627dd71a-bccc-4627-b52f-6914d4060a43-istiod-ca-cert\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.492839 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.492795    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627dd71a-bccc-4627-b52f-6914d4060a43-istio-podinfo\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.492839 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.492823    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-credential-socket\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.492839 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.492838    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7mpn\" (UniqueName: \"kubernetes.io/projected/627dd71a-bccc-4627-b52f-6914d4060a43-kube-api-access-s7mpn\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.493133 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.492895    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-workload-certs\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.493317 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.493288    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-istio-data\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.493449 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.493328    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-workload-certs\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.493449 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.493372    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-credential-socket\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.493543 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.493444    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-workload-socket\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.493807 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.493785    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627dd71a-bccc-4627-b52f-6914d4060a43-istiod-ca-cert\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.495096 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.495075    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-istio-envoy\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.495261 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.495245    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627dd71a-bccc-4627-b52f-6914d4060a43-istio-podinfo\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.501324 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.501303    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7mpn\" (UniqueName: \"kubernetes.io/projected/627dd71a-bccc-4627-b52f-6914d4060a43-kube-api-access-s7mpn\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.501592 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.501574    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627dd71a-bccc-4627-b52f-6914d4060a43-istio-token\") pod \"e2e-autores-305998-openshift-default-cc84f95f8-wxtqq\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") " pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:36.662594 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.662548    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"]
Sep 16 18:30:36.665121 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:30:36.665096    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b9a6dd_c92f_4782_973b_88d41b05bac7.slice/crio-0b9a45b2aacce3cd36d5f839ef01a58588affb708b639f7cac6cf505a912c7af WatchSource:0}: Error finding container 0b9a45b2aacce3cd36d5f839ef01a58588affb708b639f7cac6cf505a912c7af: Status 404 returned error can't find the container with id 0b9a45b2aacce3cd36d5f839ef01a58588affb708b639f7cac6cf505a912c7af
Sep 16 18:30:36.790705 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.790674    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c" event={"ID":"e8b9a6dd-c92f-4782-973b-88d41b05bac7","Type":"ContainerStarted","Data":"0b9a45b2aacce3cd36d5f839ef01a58588affb708b639f7cac6cf505a912c7af"}
Sep 16 18:30:36.791098 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.790719    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7718-041ca3" podUID="674a6211-f6cd-44da-b6af-053dbcb9f052" containerName="test-curl-7718-041ca3" containerID="cri-o://9ea34b6c3910ec0cc7b76bb7779f6bb9c2925160caaf615aff3bea253bb8b4c7" gracePeriod=2
Sep 16 18:30:36.792782 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:36.792747    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7718-041ca3\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" podUID="674a6211-f6cd-44da-b6af-053dbcb9f052" pod="opendatahub/test-curl-7718-041ca3"
Sep 16 18:30:36.882193 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:36.882142    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7718-041ca3\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" podUID="674a6211-f6cd-44da-b6af-053dbcb9f052" pod="opendatahub/test-curl-7718-041ca3"
Sep 16 18:30:36.967857 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.967833    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7718-4f4227"
Sep 16 18:30:36.969843 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:36.969816    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7718-4f4227\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" podUID="be7549cf-50d4-4440-9154-76a39dd09655" pod="opendatahub/test-curl-7718-4f4227"
Sep 16 18:30:36.997640 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.997615    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7fn6\" (UniqueName: \"kubernetes.io/projected/be7549cf-50d4-4440-9154-76a39dd09655-kube-api-access-s7fn6\") pod \"be7549cf-50d4-4440-9154-76a39dd09655\" (UID: \"be7549cf-50d4-4440-9154-76a39dd09655\") "
Sep 16 18:30:36.999720 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:36.999684    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7549cf-50d4-4440-9154-76a39dd09655-kube-api-access-s7fn6" (OuterVolumeSpecName: "kube-api-access-s7fn6") pod "be7549cf-50d4-4440-9154-76a39dd09655" (UID: "be7549cf-50d4-4440-9154-76a39dd09655"). InnerVolumeSpecName "kube-api-access-s7fn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:30:37.098723 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:37.098685    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7fn6\" (UniqueName: \"kubernetes.io/projected/be7549cf-50d4-4440-9154-76a39dd09655-kube-api-access-s7fn6\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:30:37.653405 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:37.653319    2554 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq" secret="" err="failed to sync secret cache: timed out waiting for the condition"
Sep 16 18:30:37.653405 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:37.653396    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:37.709981 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:37.709927    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-305998-openshift-default-dockercfg-bxkzs\""
Sep 16 18:30:37.797802 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:37.797770    2554 generic.go:356] "Generic (PLEG): container finished" podID="be7549cf-50d4-4440-9154-76a39dd09655" containerID="409f7db8938799b2f0f1acfe24628f10fe19eeb5e1cc2fdb51b543436de9a57c" exitCode=137
Sep 16 18:30:37.798292 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:37.797839    2554 scope.go:117] "RemoveContainer" containerID="409f7db8938799b2f0f1acfe24628f10fe19eeb5e1cc2fdb51b543436de9a57c"
Sep 16 18:30:37.798292 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:37.797952    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7718-4f4227"
Sep 16 18:30:37.802306 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:37.802208    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7718-4f4227\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" podUID="be7549cf-50d4-4440-9154-76a39dd09655" pod="opendatahub/test-curl-7718-4f4227"
Sep 16 18:30:37.810654 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:37.810626    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7718-4f4227\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" podUID="be7549cf-50d4-4440-9154-76a39dd09655" pod="opendatahub/test-curl-7718-4f4227"
Sep 16 18:30:37.946155 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:37.946117    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"]
Sep 16 18:30:38.886499 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:38.886461    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be7549cf-50d4-4440-9154-76a39dd09655" path="/var/lib/kubelet/pods/be7549cf-50d4-4440-9154-76a39dd09655/volumes"
Sep 16 18:30:38.921979 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:38.921956    2554 scope.go:117] "RemoveContainer" containerID="409f7db8938799b2f0f1acfe24628f10fe19eeb5e1cc2fdb51b543436de9a57c"
Sep 16 18:30:38.922353 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:38.922327    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409f7db8938799b2f0f1acfe24628f10fe19eeb5e1cc2fdb51b543436de9a57c\": container with ID starting with 409f7db8938799b2f0f1acfe24628f10fe19eeb5e1cc2fdb51b543436de9a57c not found: ID does not exist" containerID="409f7db8938799b2f0f1acfe24628f10fe19eeb5e1cc2fdb51b543436de9a57c"
Sep 16 18:30:38.922532 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:38.922363    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409f7db8938799b2f0f1acfe24628f10fe19eeb5e1cc2fdb51b543436de9a57c"} err="failed to get container status \"409f7db8938799b2f0f1acfe24628f10fe19eeb5e1cc2fdb51b543436de9a57c\": rpc error: code = NotFound desc = could not find container \"409f7db8938799b2f0f1acfe24628f10fe19eeb5e1cc2fdb51b543436de9a57c\": container with ID starting with 409f7db8938799b2f0f1acfe24628f10fe19eeb5e1cc2fdb51b543436de9a57c not found: ID does not exist"
Sep 16 18:30:38.923479 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:30:38.923447    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627dd71a_bccc_4627_b52f_6914d4060a43.slice/crio-6bc3a064c1f9cc7f9e6540df2e6d7c5d475e9d61640114afc29f287de351effb WatchSource:0}: Error finding container 6bc3a064c1f9cc7f9e6540df2e6d7c5d475e9d61640114afc29f287de351effb: Status 404 returned error can't find the container with id 6bc3a064c1f9cc7f9e6540df2e6d7c5d475e9d61640114afc29f287de351effb
Sep 16 18:30:38.927046 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:38.927013    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:30:38.927194 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:38.927076    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:30:38.927194 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:38.927105    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:30:39.066397 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:39.066373    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7718-041ca3"
Sep 16 18:30:39.118056 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:39.117480    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvh9r\" (UniqueName: \"kubernetes.io/projected/674a6211-f6cd-44da-b6af-053dbcb9f052-kube-api-access-pvh9r\") pod \"674a6211-f6cd-44da-b6af-053dbcb9f052\" (UID: \"674a6211-f6cd-44da-b6af-053dbcb9f052\") "
Sep 16 18:30:39.120221 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:39.120188    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674a6211-f6cd-44da-b6af-053dbcb9f052-kube-api-access-pvh9r" (OuterVolumeSpecName: "kube-api-access-pvh9r") pod "674a6211-f6cd-44da-b6af-053dbcb9f052" (UID: "674a6211-f6cd-44da-b6af-053dbcb9f052"). InnerVolumeSpecName "kube-api-access-pvh9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:30:39.218397 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:39.218365    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvh9r\" (UniqueName: \"kubernetes.io/projected/674a6211-f6cd-44da-b6af-053dbcb9f052-kube-api-access-pvh9r\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:30:39.813057 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:39.813013    2554 generic.go:356] "Generic (PLEG): container finished" podID="674a6211-f6cd-44da-b6af-053dbcb9f052" containerID="9ea34b6c3910ec0cc7b76bb7779f6bb9c2925160caaf615aff3bea253bb8b4c7" exitCode=137
Sep 16 18:30:39.813057 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:39.813055    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7718-041ca3"
Sep 16 18:30:39.813334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:39.813087    2554 scope.go:117] "RemoveContainer" containerID="9ea34b6c3910ec0cc7b76bb7779f6bb9c2925160caaf615aff3bea253bb8b4c7"
Sep 16 18:30:39.814971 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:39.814945    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c" event={"ID":"e8b9a6dd-c92f-4782-973b-88d41b05bac7","Type":"ContainerStarted","Data":"9aa9807cb05c67376f78de6db0404a32ac40896640e25ce787b2060f922347b0"}
Sep 16 18:30:39.815068 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:39.815002    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:30:39.816716 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:39.816692    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq" event={"ID":"627dd71a-bccc-4627-b52f-6914d4060a43","Type":"ContainerStarted","Data":"bf48b01c1c6beb71451e307277690f5dd85d64cfcadbc819b133defecac22ebd"}
Sep 16 18:30:39.816808 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:39.816737    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq" event={"ID":"627dd71a-bccc-4627-b52f-6914d4060a43","Type":"ContainerStarted","Data":"6bc3a064c1f9cc7f9e6540df2e6d7c5d475e9d61640114afc29f287de351effb"}
Sep 16 18:30:39.822748 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:39.822732    2554 scope.go:117] "RemoveContainer" containerID="9ea34b6c3910ec0cc7b76bb7779f6bb9c2925160caaf615aff3bea253bb8b4c7"
Sep 16 18:30:39.823012 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:39.822995    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea34b6c3910ec0cc7b76bb7779f6bb9c2925160caaf615aff3bea253bb8b4c7\": container with ID starting with 9ea34b6c3910ec0cc7b76bb7779f6bb9c2925160caaf615aff3bea253bb8b4c7 not found: ID does not exist" containerID="9ea34b6c3910ec0cc7b76bb7779f6bb9c2925160caaf615aff3bea253bb8b4c7"
Sep 16 18:30:39.823071 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:39.823019    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea34b6c3910ec0cc7b76bb7779f6bb9c2925160caaf615aff3bea253bb8b4c7"} err="failed to get container status \"9ea34b6c3910ec0cc7b76bb7779f6bb9c2925160caaf615aff3bea253bb8b4c7\": rpc error: code = NotFound desc = could not find container \"9ea34b6c3910ec0cc7b76bb7779f6bb9c2925160caaf615aff3bea253bb8b4c7\": container with ID starting with 9ea34b6c3910ec0cc7b76bb7779f6bb9c2925160caaf615aff3bea253bb8b4c7 not found: ID does not exist"
Sep 16 18:30:39.839645 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:39.839598    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c" podStartSLOduration=1.5189672490000001 podStartE2EDuration="3.839585349s" totalImagesPullingTime="2.3206181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:30:39.839066978 +0000 UTC m=+863.398234811" watchObservedRunningTime="2026-09-16 18:30:39.839585349 +0000 UTC m=+863.398753181"
Sep 16 18:30:39.841700 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:39.841675    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7718-041ca3\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" podUID="674a6211-f6cd-44da-b6af-053dbcb9f052" pod="opendatahub/test-curl-7718-041ca3"
Sep 16 18:30:39.887417 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:39.887376    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq" podStartSLOduration=3.887363975 podStartE2EDuration="3.887363975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:30:39.885986653 +0000 UTC m=+863.445154497" watchObservedRunningTime="2026-09-16 18:30:39.887363975 +0000 UTC m=+863.446531808"
Sep 16 18:30:40.302628 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.302595    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"]
Sep 16 18:30:40.307701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.307681    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"
Sep 16 18:30:40.311639 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:40.311590    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7718-041ca3\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" podUID="674a6211-f6cd-44da-b6af-053dbcb9f052" pod="opendatahub/test-curl-7718-041ca3"
Sep 16 18:30:40.311639 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.311633    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-a24d09-dockercfg-rr8mx\""
Sep 16 18:30:40.311824 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.311709    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-a24d09\""
Sep 16 18:30:40.330268 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.330208    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"]
Sep 16 18:30:40.429917 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.429885    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32667ec6-64cb-4b8f-87a2-87e7924794ed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz\" (UID: \"32667ec6-64cb-4b8f-87a2-87e7924794ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"
Sep 16 18:30:40.430065 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.429957    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppnsc\" (UniqueName: \"kubernetes.io/projected/32667ec6-64cb-4b8f-87a2-87e7924794ed-kube-api-access-ppnsc\") pod \"payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz\" (UID: \"32667ec6-64cb-4b8f-87a2-87e7924794ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"
Sep 16 18:30:40.530920 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.530887    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32667ec6-64cb-4b8f-87a2-87e7924794ed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz\" (UID: \"32667ec6-64cb-4b8f-87a2-87e7924794ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"
Sep 16 18:30:40.531086 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.530949    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppnsc\" (UniqueName: \"kubernetes.io/projected/32667ec6-64cb-4b8f-87a2-87e7924794ed-kube-api-access-ppnsc\") pod \"payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz\" (UID: \"32667ec6-64cb-4b8f-87a2-87e7924794ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"
Sep 16 18:30:40.531667 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.531647    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32667ec6-64cb-4b8f-87a2-87e7924794ed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz\" (UID: \"32667ec6-64cb-4b8f-87a2-87e7924794ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"
Sep 16 18:30:40.539966 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.539942    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppnsc\" (UniqueName: \"kubernetes.io/projected/32667ec6-64cb-4b8f-87a2-87e7924794ed-kube-api-access-ppnsc\") pod \"payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz\" (UID: \"32667ec6-64cb-4b8f-87a2-87e7924794ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"
Sep 16 18:30:40.620932 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.620850    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"
Sep 16 18:30:40.654211 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.654176    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:40.659478 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.659453    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:40.661555 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:40.661520    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7718-041ca3\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" podUID="674a6211-f6cd-44da-b6af-053dbcb9f052" pod="opendatahub/test-curl-7718-041ca3"
Sep 16 18:30:40.822536 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.822501    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:40.823324 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.823304    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:30:40.827040 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:30:40.827020    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7718-041ca3\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" podUID="674a6211-f6cd-44da-b6af-053dbcb9f052" pod="opendatahub/test-curl-7718-041ca3"
Sep 16 18:30:40.885164 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.885091    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="674a6211-f6cd-44da-b6af-053dbcb9f052" path="/var/lib/kubelet/pods/674a6211-f6cd-44da-b6af-053dbcb9f052/volumes"
Sep 16 18:30:40.936965 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:40.936940    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"]
Sep 16 18:30:40.938843 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:30:40.938818    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32667ec6_64cb_4b8f_87a2_87e7924794ed.slice/crio-81156108db343f2f42d1d49380ba49a887ccdb1c67394f76bab735cf55d12bb8 WatchSource:0}: Error finding container 81156108db343f2f42d1d49380ba49a887ccdb1c67394f76bab735cf55d12bb8: Status 404 returned error can't find the container with id 81156108db343f2f42d1d49380ba49a887ccdb1c67394f76bab735cf55d12bb8
Sep 16 18:30:41.830172 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:41.830130    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz" event={"ID":"32667ec6-64cb-4b8f-87a2-87e7924794ed","Type":"ContainerStarted","Data":"81156108db343f2f42d1d49380ba49a887ccdb1c67394f76bab735cf55d12bb8"}
Sep 16 18:30:43.842168 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:43.842124    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz" event={"ID":"32667ec6-64cb-4b8f-87a2-87e7924794ed","Type":"ContainerStarted","Data":"938ae6aea147472e421fbde33946611826bc56a19c7ca3ce893c8e066208eb91"}
Sep 16 18:30:43.842679 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:43.842221    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"
Sep 16 18:30:43.867409 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:43.867354    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz" podStartSLOduration=1.642412411 podStartE2EDuration="3.867339486s" totalImagesPullingTime="2.224927075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:30:43.8661573 +0000 UTC m=+867.425325144" watchObservedRunningTime="2026-09-16 18:30:43.867339486 +0000 UTC m=+867.426507318"
Sep 16 18:30:45.828331 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:45.828299    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:30:46.210497 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:46.210406    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:30:47.259983 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.259950    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"]
Sep 16 18:30:47.264028 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.264005    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.266194 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.266171    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5ca9c2-openshift-default-dockercfg-d2t2k\""
Sep 16 18:30:47.282136 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.282099    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"]
Sep 16 18:30:47.399312 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.399273    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-workload-socket\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.399312 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.399316    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-credential-socket\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.399538 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.399336    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-istio-envoy\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.399538 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.399358    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-workload-certs\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.399538 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.399406    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f794beb-3530-46d7-a892-b5e59344638c-istio-token\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.399538 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.399503    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzgxz\" (UniqueName: \"kubernetes.io/projected/1f794beb-3530-46d7-a892-b5e59344638c-kube-api-access-zzgxz\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.399669 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.399640    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-istio-data\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.399707 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.399674    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f794beb-3530-46d7-a892-b5e59344638c-istiod-ca-cert\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.399763 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.399744    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f794beb-3530-46d7-a892-b5e59344638c-istio-podinfo\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.501102 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.501057    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-istio-data\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.501102 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.501103    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f794beb-3530-46d7-a892-b5e59344638c-istiod-ca-cert\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.501378 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.501124    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f794beb-3530-46d7-a892-b5e59344638c-istio-podinfo\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.501378 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.501201    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-workload-socket\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.501378 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.501252    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-credential-socket\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.501378 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.501284    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-istio-envoy\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.501378 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.501307    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-workload-certs\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.501378 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.501328    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f794beb-3530-46d7-a892-b5e59344638c-istio-token\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.501378 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.501360    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzgxz\" (UniqueName: \"kubernetes.io/projected/1f794beb-3530-46d7-a892-b5e59344638c-kube-api-access-zzgxz\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.501734 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.501583    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-istio-data\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.501734 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.501676    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-workload-socket\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.501838 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.501733    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-credential-socket\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.502015 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.501993    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f794beb-3530-46d7-a892-b5e59344638c-istiod-ca-cert\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.502090 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.502012    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-workload-certs\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.503730 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.503706    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-istio-envoy\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.503854 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.503745    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f794beb-3530-46d7-a892-b5e59344638c-istio-podinfo\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.509068 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.509046    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzgxz\" (UniqueName: \"kubernetes.io/projected/1f794beb-3530-46d7-a892-b5e59344638c-kube-api-access-zzgxz\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.509669 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.509651    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f794beb-3530-46d7-a892-b5e59344638c-istio-token\") pod \"e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") " pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.575161 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.575067    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:47.894975 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:30:47.894939    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f794beb_3530_46d7_a892_b5e59344638c.slice/crio-138dedd61560833fee6a2999f574b4879081759e874f978b4b611ffa8532d46c WatchSource:0}: Error finding container 138dedd61560833fee6a2999f574b4879081759e874f978b4b611ffa8532d46c: Status 404 returned error can't find the container with id 138dedd61560833fee6a2999f574b4879081759e874f978b4b611ffa8532d46c
Sep 16 18:30:47.895476 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.895453    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"]
Sep 16 18:30:47.898712 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.898679    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:30:47.898772 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.898746    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:30:47.898805 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:47.898772    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:30:48.870335 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:48.870299    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf" event={"ID":"1f794beb-3530-46d7-a892-b5e59344638c","Type":"ContainerStarted","Data":"bfb3cf25e2cdd2bbe0e5c105577f1cbb57db5965bfa6b69d5d4f7f813fb0ed60"}
Sep 16 18:30:48.870730 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:48.870383    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf" event={"ID":"1f794beb-3530-46d7-a892-b5e59344638c","Type":"ContainerStarted","Data":"138dedd61560833fee6a2999f574b4879081759e874f978b4b611ffa8532d46c"}
Sep 16 18:30:48.892535 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:48.892486    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf" podStartSLOduration=1.892464288 podStartE2EDuration="1.892464288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:30:48.890773184 +0000 UTC m=+872.449941020" watchObservedRunningTime="2026-09-16 18:30:48.892464288 +0000 UTC m=+872.451632121"
Sep 16 18:30:49.439070 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:49.439031    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"]
Sep 16 18:30:49.442656 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:49.442632    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"
Sep 16 18:30:49.444758 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:49.444733    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-78d09e99\""
Sep 16 18:30:49.444917 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:49.444895    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-78d09e99-dockercfg-57t96\""
Sep 16 18:30:49.452340 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:49.452278    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"]
Sep 16 18:30:49.520932 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:49.520892    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2139effc-062a-45b6-9946-0aa4af823fa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft\" (UID: \"2139effc-062a-45b6-9946-0aa4af823fa3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"
Sep 16 18:30:49.521336 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:49.520957    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-544vq\" (UniqueName: \"kubernetes.io/projected/2139effc-062a-45b6-9946-0aa4af823fa3-kube-api-access-544vq\") pod \"payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft\" (UID: \"2139effc-062a-45b6-9946-0aa4af823fa3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"
Sep 16 18:30:49.576206 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:49.576174    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:49.581418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:49.581389    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:49.621690 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:49.621650    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2139effc-062a-45b6-9946-0aa4af823fa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft\" (UID: \"2139effc-062a-45b6-9946-0aa4af823fa3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"
Sep 16 18:30:49.621835 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:49.621701    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544vq\" (UniqueName: \"kubernetes.io/projected/2139effc-062a-45b6-9946-0aa4af823fa3-kube-api-access-544vq\") pod \"payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft\" (UID: \"2139effc-062a-45b6-9946-0aa4af823fa3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"
Sep 16 18:30:49.622385 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:49.622367    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2139effc-062a-45b6-9946-0aa4af823fa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft\" (UID: \"2139effc-062a-45b6-9946-0aa4af823fa3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"
Sep 16 18:30:49.629877 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:49.629853    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-544vq\" (UniqueName: \"kubernetes.io/projected/2139effc-062a-45b6-9946-0aa4af823fa3-kube-api-access-544vq\") pod \"payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft\" (UID: \"2139effc-062a-45b6-9946-0aa4af823fa3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"
Sep 16 18:30:49.756288 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:49.756251    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"
Sep 16 18:30:49.879154 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:49.879114    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:49.880279 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:49.880257    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:30:50.042446 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:50.041225    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"]
Sep 16 18:30:50.886327 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:50.886263    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft" event={"ID":"2139effc-062a-45b6-9946-0aa4af823fa3","Type":"ContainerStarted","Data":"9f0ef9a8bab3bb2aab1789dbc2498ae73b79637d2e73cd3d340323698e5d6800"}
Sep 16 18:30:50.886327 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:50.886326    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft" event={"ID":"2139effc-062a-45b6-9946-0aa4af823fa3","Type":"ContainerStarted","Data":"84a5fd0fddbe60bbd2ac171560132a4fb7e189cea06d02688f7cc3450b6db654"}
Sep 16 18:30:50.886812 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:50.886429    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"
Sep 16 18:30:50.903647 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:50.903591    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft" podStartSLOduration=1.9035757979999999 podStartE2EDuration="1.903575798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:30:50.902151905 +0000 UTC m=+874.461319752" watchObservedRunningTime="2026-09-16 18:30:50.903575798 +0000 UTC m=+874.462743630"
Sep 16 18:30:54.851425 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:54.851391    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"
Sep 16 18:30:56.393019 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.392980    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"]
Sep 16 18:30:56.397254 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.397219    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:30:56.399353 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.399328    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5ca9c2-serving-cert\""
Sep 16 18:30:56.407818 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.407799    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"]
Sep 16 18:30:56.500447 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.500394    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plg5j\" (UniqueName: \"kubernetes.io/projected/8e2e1faf-016d-4bfb-be40-85eb52662ca0-kube-api-access-plg5j\") pod \"maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p\" (UID: \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:30:56.500675 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.500504    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e2e1faf-016d-4bfb-be40-85eb52662ca0-maas-api-tls\") pod \"maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p\" (UID: \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:30:56.500675 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.500572    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2e1faf-016d-4bfb-be40-85eb52662ca0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p\" (UID: \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:30:56.500815 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.500694    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e2e1faf-016d-4bfb-be40-85eb52662ca0-metrics-certs\") pod \"maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p\" (UID: \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:30:56.601513 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.601465    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plg5j\" (UniqueName: \"kubernetes.io/projected/8e2e1faf-016d-4bfb-be40-85eb52662ca0-kube-api-access-plg5j\") pod \"maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p\" (UID: \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:30:56.601513 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.601511    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e2e1faf-016d-4bfb-be40-85eb52662ca0-maas-api-tls\") pod \"maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p\" (UID: \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:30:56.601801 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.601686    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2e1faf-016d-4bfb-be40-85eb52662ca0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p\" (UID: \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:30:56.601801 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.601761    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e2e1faf-016d-4bfb-be40-85eb52662ca0-metrics-certs\") pod \"maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p\" (UID: \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:30:56.602382 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.602360    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2e1faf-016d-4bfb-be40-85eb52662ca0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p\" (UID: \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:30:56.603891 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.603865    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e2e1faf-016d-4bfb-be40-85eb52662ca0-maas-api-tls\") pod \"maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p\" (UID: \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:30:56.604005 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.603989    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e2e1faf-016d-4bfb-be40-85eb52662ca0-metrics-certs\") pod \"maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p\" (UID: \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:30:56.610495 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.610470    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plg5j\" (UniqueName: \"kubernetes.io/projected/8e2e1faf-016d-4bfb-be40-85eb52662ca0-kube-api-access-plg5j\") pod \"maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p\" (UID: \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:30:56.712723 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.712639    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:30:56.991553 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:56.991528    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"]
Sep 16 18:30:56.993904 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:30:56.993878    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2e1faf_016d_4bfb_be40_85eb52662ca0.slice/crio-ec4cf3304b37d26ce893c6396e519478c001d2e9c6188be1869179d9576b6ac1 WatchSource:0}: Error finding container ec4cf3304b37d26ce893c6396e519478c001d2e9c6188be1869179d9576b6ac1: Status 404 returned error can't find the container with id ec4cf3304b37d26ce893c6396e519478c001d2e9c6188be1869179d9576b6ac1
Sep 16 18:30:57.035061 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.035027    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"]
Sep 16 18:30:57.040116 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.040094    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"
Sep 16 18:30:57.041684 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.041654    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"]
Sep 16 18:30:57.042511 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.042493    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5ca9c2-dockercfg-tzdjc\""
Sep 16 18:30:57.042929 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.042902    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5ca9c2\""
Sep 16 18:30:57.047258 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.046813    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"
Sep 16 18:30:57.052672 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.052638    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"]
Sep 16 18:30:57.055860 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.055798    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"]
Sep 16 18:30:57.208142 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.208112    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd01819-79e8-4316-83db-dda60ac5aeb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7\" (UID: \"afd01819-79e8-4316-83db-dda60ac5aeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"
Sep 16 18:30:57.208335 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.208157    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6db1ae7-cf0b-4760-b50c-a1733be0e781-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz\" (UID: \"c6db1ae7-cf0b-4760-b50c-a1733be0e781\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"
Sep 16 18:30:57.208335 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.208183    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd696\" (UniqueName: \"kubernetes.io/projected/c6db1ae7-cf0b-4760-b50c-a1733be0e781-kube-api-access-rd696\") pod \"payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz\" (UID: \"c6db1ae7-cf0b-4760-b50c-a1733be0e781\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"
Sep 16 18:30:57.208335 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.208310    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5948h\" (UniqueName: \"kubernetes.io/projected/afd01819-79e8-4316-83db-dda60ac5aeb3-kube-api-access-5948h\") pod \"payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7\" (UID: \"afd01819-79e8-4316-83db-dda60ac5aeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"
Sep 16 18:30:57.310037 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.309948    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5948h\" (UniqueName: \"kubernetes.io/projected/afd01819-79e8-4316-83db-dda60ac5aeb3-kube-api-access-5948h\") pod \"payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7\" (UID: \"afd01819-79e8-4316-83db-dda60ac5aeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"
Sep 16 18:30:57.310182 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.310045    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd01819-79e8-4316-83db-dda60ac5aeb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7\" (UID: \"afd01819-79e8-4316-83db-dda60ac5aeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"
Sep 16 18:30:57.310182 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.310095    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6db1ae7-cf0b-4760-b50c-a1733be0e781-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz\" (UID: \"c6db1ae7-cf0b-4760-b50c-a1733be0e781\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"
Sep 16 18:30:57.310182 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.310123    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd696\" (UniqueName: \"kubernetes.io/projected/c6db1ae7-cf0b-4760-b50c-a1733be0e781-kube-api-access-rd696\") pod \"payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz\" (UID: \"c6db1ae7-cf0b-4760-b50c-a1733be0e781\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"
Sep 16 18:30:57.310958 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.310935    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd01819-79e8-4316-83db-dda60ac5aeb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7\" (UID: \"afd01819-79e8-4316-83db-dda60ac5aeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"
Sep 16 18:30:57.310992 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.310945    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6db1ae7-cf0b-4760-b50c-a1733be0e781-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz\" (UID: \"c6db1ae7-cf0b-4760-b50c-a1733be0e781\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"
Sep 16 18:30:57.317814 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.317789    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5948h\" (UniqueName: \"kubernetes.io/projected/afd01819-79e8-4316-83db-dda60ac5aeb3-kube-api-access-5948h\") pod \"payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7\" (UID: \"afd01819-79e8-4316-83db-dda60ac5aeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"
Sep 16 18:30:57.318064 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.318045    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd696\" (UniqueName: \"kubernetes.io/projected/c6db1ae7-cf0b-4760-b50c-a1733be0e781-kube-api-access-rd696\") pod \"payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz\" (UID: \"c6db1ae7-cf0b-4760-b50c-a1733be0e781\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"
Sep 16 18:30:57.356787 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.356758    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"
Sep 16 18:30:57.363115 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.363095    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"
Sep 16 18:30:57.637153 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.637123    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"]
Sep 16 18:30:57.639102 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:30:57.639073    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd01819_79e8_4316_83db_dda60ac5aeb3.slice/crio-d795bd7999829a64ba48d3e54346130232610d2b81d5ce43969407bec9938db2 WatchSource:0}: Error finding container d795bd7999829a64ba48d3e54346130232610d2b81d5ce43969407bec9938db2: Status 404 returned error can't find the container with id d795bd7999829a64ba48d3e54346130232610d2b81d5ce43969407bec9938db2
Sep 16 18:30:57.647026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.645075    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"]
Sep 16 18:30:57.651485 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:30:57.651458    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6db1ae7_cf0b_4760_b50c_a1733be0e781.slice/crio-6157bec46fc6309cfdb3cad869abf65bdf605a56d0b73d21f149dce66c634027 WatchSource:0}: Error finding container 6157bec46fc6309cfdb3cad869abf65bdf605a56d0b73d21f149dce66c634027: Status 404 returned error can't find the container with id 6157bec46fc6309cfdb3cad869abf65bdf605a56d0b73d21f149dce66c634027
Sep 16 18:30:57.925566 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.925480    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz" event={"ID":"c6db1ae7-cf0b-4760-b50c-a1733be0e781","Type":"ContainerStarted","Data":"e5cc9d0ad716ba50d19e55bc725b8377f826e7f5d8045f1c3e581eadaf6e3ab7"}
Sep 16 18:30:57.925566 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.925536    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz" event={"ID":"c6db1ae7-cf0b-4760-b50c-a1733be0e781","Type":"ContainerStarted","Data":"6157bec46fc6309cfdb3cad869abf65bdf605a56d0b73d21f149dce66c634027"}
Sep 16 18:30:57.925785 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.925601    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"
Sep 16 18:30:57.927349 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.927321    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p" event={"ID":"8e2e1faf-016d-4bfb-be40-85eb52662ca0","Type":"ContainerStarted","Data":"10fb28e142ff79f6ecb396efa35e493da2e112399ea52d896e30969ee2f7b33c"}
Sep 16 18:30:57.927465 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.927370    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p" event={"ID":"8e2e1faf-016d-4bfb-be40-85eb52662ca0","Type":"ContainerStarted","Data":"ec4cf3304b37d26ce893c6396e519478c001d2e9c6188be1869179d9576b6ac1"}
Sep 16 18:30:57.927465 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.927414    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:30:57.929146 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.929123    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7" event={"ID":"afd01819-79e8-4316-83db-dda60ac5aeb3","Type":"ContainerStarted","Data":"b9dac7c897371e4c1e0a9d412f95df6242d81448ebe837a21ec26925ddddf2b5"}
Sep 16 18:30:57.929285 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.929167    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7" event={"ID":"afd01819-79e8-4316-83db-dda60ac5aeb3","Type":"ContainerStarted","Data":"d795bd7999829a64ba48d3e54346130232610d2b81d5ce43969407bec9938db2"}
Sep 16 18:30:57.929335 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.929301    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"
Sep 16 18:30:57.945289 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.945222    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz" podStartSLOduration=0.945203486 podStartE2EDuration="945.203486ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:30:57.944126536 +0000 UTC m=+881.503294370" watchObservedRunningTime="2026-09-16 18:30:57.945203486 +0000 UTC m=+881.504371320"
Sep 16 18:30:57.965384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.965322    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7" podStartSLOduration=0.965299708 podStartE2EDuration="965.299708ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:30:57.965007664 +0000 UTC m=+881.524175499" watchObservedRunningTime="2026-09-16 18:30:57.965299708 +0000 UTC m=+881.524467542"
Sep 16 18:30:57.986609 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:57.986546    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p" podStartSLOduration=1.6573299320000001 podStartE2EDuration="1.986523881s" totalImagesPullingTime="329.193949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:30:57.983046451 +0000 UTC m=+881.542214323" watchObservedRunningTime="2026-09-16 18:30:57.986523881 +0000 UTC m=+881.545691714"
Sep 16 18:30:58.936077 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:58.935980    2554 generic.go:356] "Generic (PLEG): container finished" podID="335a6bc4-c505-4fab-b12f-d2ccc04e971d" containerID="6f267a416e3af69e03dbdc905a8b756371124c27fe044f0c01a68b5410bb2668" exitCode=0
Sep 16 18:30:58.936077 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:58.936062    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm" event={"ID":"335a6bc4-c505-4fab-b12f-d2ccc04e971d","Type":"ContainerDied","Data":"6f267a416e3af69e03dbdc905a8b756371124c27fe044f0c01a68b5410bb2668"}
Sep 16 18:30:58.936644 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:58.936626    2554 scope.go:117] "RemoveContainer" containerID="6f267a416e3af69e03dbdc905a8b756371124c27fe044f0c01a68b5410bb2668"
Sep 16 18:30:58.937753 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:58.937721    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:30:58.937855 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:58.937803    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:30:58.937916 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:58.937868    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:30:59.945076 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:30:59.945033    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm" event={"ID":"335a6bc4-c505-4fab-b12f-d2ccc04e971d","Type":"ContainerStarted","Data":"7de6264a99c2038d6575467655c9edbd7b05198e1f70c231a708edeab9b3244e"}
Sep 16 18:31:00.375732 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:00.375686    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:31:00.376814 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:00.376775    2554 patch_prober.go:28] interesting pod/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 16 18:31:00.377057 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:00.377011    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm" podUID="335a6bc4-c505-4fab-b12f-d2ccc04e971d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 16 18:31:01.375705 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:01.375656    2554 patch_prober.go:28] interesting pod/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 16 18:31:01.376207 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:01.375731    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm" podUID="335a6bc4-c505-4fab-b12f-d2ccc04e971d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 16 18:31:01.893881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:01.893843    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"
Sep 16 18:31:02.376580 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:02.376542    2554 patch_prober.go:28] interesting pod/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 16 18:31:02.377176 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:02.377127    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm" podUID="335a6bc4-c505-4fab-b12f-d2ccc04e971d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 16 18:31:03.376152 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:03.376115    2554 patch_prober.go:28] interesting pod/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 16 18:31:03.376513 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:03.376454    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm" podUID="335a6bc4-c505-4fab-b12f-d2ccc04e971d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 16 18:31:03.769251 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:03.769204    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:31:03.953400 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:03.953368    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:31:04.376491 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:04.376439    2554 patch_prober.go:28] interesting pod/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 16 18:31:04.376710 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:04.376518    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm" podUID="335a6bc4-c505-4fab-b12f-d2ccc04e971d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 16 18:31:05.380892 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:05.380818    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:31:05.381896 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:05.381874    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:31:08.938534 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:08.938496    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"
Sep 16 18:31:08.939030 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:08.938582    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"
Sep 16 18:31:09.037921 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:09.037877    2554 generic.go:356] "Generic (PLEG): container finished" podID="627dd71a-bccc-4627-b52f-6914d4060a43" containerID="bf48b01c1c6beb71451e307277690f5dd85d64cfcadbc819b133defecac22ebd" exitCode=0
Sep 16 18:31:09.038092 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:09.037923    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq" event={"ID":"627dd71a-bccc-4627-b52f-6914d4060a43","Type":"ContainerDied","Data":"bf48b01c1c6beb71451e307277690f5dd85d64cfcadbc819b133defecac22ebd"}
Sep 16 18:31:09.038387 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:09.038372    2554 scope.go:117] "RemoveContainer" containerID="bf48b01c1c6beb71451e307277690f5dd85d64cfcadbc819b133defecac22ebd"
Sep 16 18:31:09.039522 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:09.039494    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:31:09.039584 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:09.039562    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:31:09.039620 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:09.039596    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:31:09.091109 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:09.091078    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:31:10.049490 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:10.048025    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq" event={"ID":"627dd71a-bccc-4627-b52f-6914d4060a43","Type":"ContainerStarted","Data":"1ee059c5d7867b86f5d13d9c252ce7e18661829e9f86bf1ab63cff6d9bed3da1"}
Sep 16 18:31:10.655029 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:10.654868    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:31:10.663263 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:10.655918    2554 patch_prober.go:28] interesting pod/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 16 18:31:10.663263 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:10.655998    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq" podUID="627dd71a-bccc-4627-b52f-6914d4060a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 16 18:31:10.822757 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:10.822714    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:31:11.654200 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:11.654159    2554 patch_prober.go:28] interesting pod/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 16 18:31:11.654798 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:11.654762    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq" podUID="627dd71a-bccc-4627-b52f-6914d4060a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 16 18:31:12.654996 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:12.654952    2554 patch_prober.go:28] interesting pod/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 16 18:31:12.655508 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:12.655043    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq" podUID="627dd71a-bccc-4627-b52f-6914d4060a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 16 18:31:13.656026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:13.655574    2554 patch_prober.go:28] interesting pod/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 16 18:31:13.656026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:13.655651    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq" podUID="627dd71a-bccc-4627-b52f-6914d4060a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 16 18:31:14.654374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:14.654330    2554 patch_prober.go:28] interesting pod/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Sep 16 18:31:14.654598 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:14.654400    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq" podUID="627dd71a-bccc-4627-b52f-6914d4060a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Sep 16 18:31:15.662171 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:15.662131    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:31:15.664674 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:15.664643    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:31:16.826662 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:16.826633    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:31:16.827557 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:16.827538    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:31:19.880876 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:19.880838    2554 patch_prober.go:28] interesting pod/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 16 18:31:19.881324 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:19.880894    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf" podUID="1f794beb-3530-46d7-a892-b5e59344638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 16 18:31:20.116051 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:20.116017    2554 generic.go:356] "Generic (PLEG): container finished" podID="1f794beb-3530-46d7-a892-b5e59344638c" containerID="bfb3cf25e2cdd2bbe0e5c105577f1cbb57db5965bfa6b69d5d4f7f813fb0ed60" exitCode=0
Sep 16 18:31:20.116216 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:20.116091    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf" event={"ID":"1f794beb-3530-46d7-a892-b5e59344638c","Type":"ContainerDied","Data":"bfb3cf25e2cdd2bbe0e5c105577f1cbb57db5965bfa6b69d5d4f7f813fb0ed60"}
Sep 16 18:31:20.116587 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:20.116572    2554 scope.go:117] "RemoveContainer" containerID="bfb3cf25e2cdd2bbe0e5c105577f1cbb57db5965bfa6b69d5d4f7f813fb0ed60"
Sep 16 18:31:20.117632 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:20.117601    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:31:20.117730 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:20.117682    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:31:20.117792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:20.117726    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:31:21.123466 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:21.123429    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf" event={"ID":"1f794beb-3530-46d7-a892-b5e59344638c","Type":"ContainerStarted","Data":"e90183f4c31856a36e9c953241db9e10c3c305224907ca4bfd5ff153e835327f"}
Sep 16 18:31:21.576190 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:21.576147    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:31:21.576452 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:21.576424    2554 patch_prober.go:28] interesting pod/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 16 18:31:21.576521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:21.576498    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf" podUID="1f794beb-3530-46d7-a892-b5e59344638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 16 18:31:22.575599 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:22.575562    2554 patch_prober.go:28] interesting pod/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 16 18:31:22.575971 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:22.575621    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf" podUID="1f794beb-3530-46d7-a892-b5e59344638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 16 18:31:23.575928 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:23.575893    2554 patch_prober.go:28] interesting pod/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 16 18:31:23.576347 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:23.575952    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf" podUID="1f794beb-3530-46d7-a892-b5e59344638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 16 18:31:24.400155 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:24.400118    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"]
Sep 16 18:31:24.400434 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:24.400390    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft" podUID="2139effc-062a-45b6-9946-0aa4af823fa3" containerName="payload-pre-processing" containerID="cri-o://9f0ef9a8bab3bb2aab1789dbc2498ae73b79637d2e73cd3d340323698e5d6800" gracePeriod=30
Sep 16 18:31:24.575851 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:24.575812    2554 patch_prober.go:28] interesting pod/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 16 18:31:24.576017 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:24.575877    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf" podUID="1f794beb-3530-46d7-a892-b5e59344638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 16 18:31:24.705596 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:24.705565    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"
Sep 16 18:31:24.803343 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:24.803314    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2139effc-062a-45b6-9946-0aa4af823fa3-plugins-config-volume\") pod \"2139effc-062a-45b6-9946-0aa4af823fa3\" (UID: \"2139effc-062a-45b6-9946-0aa4af823fa3\") "
Sep 16 18:31:24.803550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:24.803392    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-544vq\" (UniqueName: \"kubernetes.io/projected/2139effc-062a-45b6-9946-0aa4af823fa3-kube-api-access-544vq\") pod \"2139effc-062a-45b6-9946-0aa4af823fa3\" (UID: \"2139effc-062a-45b6-9946-0aa4af823fa3\") "
Sep 16 18:31:24.803769 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:24.803748    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2139effc-062a-45b6-9946-0aa4af823fa3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2139effc-062a-45b6-9946-0aa4af823fa3" (UID: "2139effc-062a-45b6-9946-0aa4af823fa3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:31:24.805680 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:24.805650    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2139effc-062a-45b6-9946-0aa4af823fa3-kube-api-access-544vq" (OuterVolumeSpecName: "kube-api-access-544vq") pod "2139effc-062a-45b6-9946-0aa4af823fa3" (UID: "2139effc-062a-45b6-9946-0aa4af823fa3"). InnerVolumeSpecName "kube-api-access-544vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:31:24.904968 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:24.904941    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2139effc-062a-45b6-9946-0aa4af823fa3-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:31:24.904968 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:24.904966    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-544vq\" (UniqueName: \"kubernetes.io/projected/2139effc-062a-45b6-9946-0aa4af823fa3-kube-api-access-544vq\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:31:25.154690 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:25.152256    2554 generic.go:356] "Generic (PLEG): container finished" podID="2139effc-062a-45b6-9946-0aa4af823fa3" containerID="9f0ef9a8bab3bb2aab1789dbc2498ae73b79637d2e73cd3d340323698e5d6800" exitCode=0
Sep 16 18:31:25.154690 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:25.152317    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft" event={"ID":"2139effc-062a-45b6-9946-0aa4af823fa3","Type":"ContainerDied","Data":"9f0ef9a8bab3bb2aab1789dbc2498ae73b79637d2e73cd3d340323698e5d6800"}
Sep 16 18:31:25.154690 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:25.152367    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft" event={"ID":"2139effc-062a-45b6-9946-0aa4af823fa3","Type":"ContainerDied","Data":"84a5fd0fddbe60bbd2ac171560132a4fb7e189cea06d02688f7cc3450b6db654"}
Sep 16 18:31:25.154690 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:25.152398    2554 scope.go:117] "RemoveContainer" containerID="9f0ef9a8bab3bb2aab1789dbc2498ae73b79637d2e73cd3d340323698e5d6800"
Sep 16 18:31:25.154690 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:25.152562    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"
Sep 16 18:31:25.176055 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:25.175774    2554 scope.go:117] "RemoveContainer" containerID="9f0ef9a8bab3bb2aab1789dbc2498ae73b79637d2e73cd3d340323698e5d6800"
Sep 16 18:31:25.176252 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:31:25.176203    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0ef9a8bab3bb2aab1789dbc2498ae73b79637d2e73cd3d340323698e5d6800\": container with ID starting with 9f0ef9a8bab3bb2aab1789dbc2498ae73b79637d2e73cd3d340323698e5d6800 not found: ID does not exist" containerID="9f0ef9a8bab3bb2aab1789dbc2498ae73b79637d2e73cd3d340323698e5d6800"
Sep 16 18:31:25.176338 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:25.176255    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0ef9a8bab3bb2aab1789dbc2498ae73b79637d2e73cd3d340323698e5d6800"} err="failed to get container status \"9f0ef9a8bab3bb2aab1789dbc2498ae73b79637d2e73cd3d340323698e5d6800\": rpc error: code = NotFound desc = could not find container \"9f0ef9a8bab3bb2aab1789dbc2498ae73b79637d2e73cd3d340323698e5d6800\": container with ID starting with 9f0ef9a8bab3bb2aab1789dbc2498ae73b79637d2e73cd3d340323698e5d6800 not found: ID does not exist"
Sep 16 18:31:25.192240 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:25.192209    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"]
Sep 16 18:31:25.201803 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:25.201775    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-78d09e99-54f955b57d-shxft"]
Sep 16 18:31:25.576299 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:25.576263    2554 patch_prober.go:28] interesting pod/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 16 18:31:25.576712 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:25.576318    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf" podUID="1f794beb-3530-46d7-a892-b5e59344638c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 16 18:31:26.579831 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:26.579796    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:31:26.580327 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:26.580182    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:31:26.580748 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:26.580729    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:31:26.886124 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:26.886045    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2139effc-062a-45b6-9946-0aa4af823fa3" path="/var/lib/kubelet/pods/2139effc-062a-45b6-9946-0aa4af823fa3/volumes"
Sep 16 18:31:28.600157 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:28.600122    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:31:30.682482 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:30.682447    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"]
Sep 16 18:31:30.682923 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:30.682909    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2139effc-062a-45b6-9946-0aa4af823fa3" containerName="payload-pre-processing"
Sep 16 18:31:30.682967 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:30.682924    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2139effc-062a-45b6-9946-0aa4af823fa3" containerName="payload-pre-processing"
Sep 16 18:31:30.683009 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:30.683000    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2139effc-062a-45b6-9946-0aa4af823fa3" containerName="payload-pre-processing"
Sep 16 18:31:30.685621 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:30.685603    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"
Sep 16 18:31:30.687963 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:30.687940    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-78aa4077\""
Sep 16 18:31:30.688095 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:30.687942    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-78aa4077-dockercfg-c89fm\""
Sep 16 18:31:30.695564 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:30.695527    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"]
Sep 16 18:31:30.763809 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:30.763774    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm2r7\" (UniqueName: \"kubernetes.io/projected/baa836b3-5375-4d64-ae4e-384f4e7b21c5-kube-api-access-pm2r7\") pod \"payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx\" (UID: \"baa836b3-5375-4d64-ae4e-384f4e7b21c5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"
Sep 16 18:31:30.764337 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:30.763912    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa836b3-5375-4d64-ae4e-384f4e7b21c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx\" (UID: \"baa836b3-5375-4d64-ae4e-384f4e7b21c5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"
Sep 16 18:31:30.864960 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:30.864922    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa836b3-5375-4d64-ae4e-384f4e7b21c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx\" (UID: \"baa836b3-5375-4d64-ae4e-384f4e7b21c5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"
Sep 16 18:31:30.865162 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:30.865024    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm2r7\" (UniqueName: \"kubernetes.io/projected/baa836b3-5375-4d64-ae4e-384f4e7b21c5-kube-api-access-pm2r7\") pod \"payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx\" (UID: \"baa836b3-5375-4d64-ae4e-384f4e7b21c5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"
Sep 16 18:31:30.865666 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:30.865643    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa836b3-5375-4d64-ae4e-384f4e7b21c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx\" (UID: \"baa836b3-5375-4d64-ae4e-384f4e7b21c5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"
Sep 16 18:31:30.874035 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:30.874005    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm2r7\" (UniqueName: \"kubernetes.io/projected/baa836b3-5375-4d64-ae4e-384f4e7b21c5-kube-api-access-pm2r7\") pod \"payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx\" (UID: \"baa836b3-5375-4d64-ae4e-384f4e7b21c5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"
Sep 16 18:31:30.996858 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:30.996819    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"
Sep 16 18:31:31.269876 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:31.269849    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"]
Sep 16 18:31:31.272120 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:31:31.272095    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa836b3_5375_4d64_ae4e_384f4e7b21c5.slice/crio-a32b705ed3740ccd52473e3e6e071f36b62597b417b8b4a51f08a842447b7bdd WatchSource:0}: Error finding container a32b705ed3740ccd52473e3e6e071f36b62597b417b8b4a51f08a842447b7bdd: Status 404 returned error can't find the container with id a32b705ed3740ccd52473e3e6e071f36b62597b417b8b4a51f08a842447b7bdd
Sep 16 18:31:32.194651 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:32.194612    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx" event={"ID":"baa836b3-5375-4d64-ae4e-384f4e7b21c5","Type":"ContainerStarted","Data":"df9becdae0b16706aae0ec8ded97ae6e4d3a71cb62ed5e4eac2727a76a47618e"}
Sep 16 18:31:32.195036 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:32.194676    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx" event={"ID":"baa836b3-5375-4d64-ae4e-384f4e7b21c5","Type":"ContainerStarted","Data":"a32b705ed3740ccd52473e3e6e071f36b62597b417b8b4a51f08a842447b7bdd"}
Sep 16 18:31:32.195036 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:32.194828    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"
Sep 16 18:31:32.211114 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:32.211073    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx" podStartSLOduration=2.211059072 podStartE2EDuration="2.211059072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:31:32.20870638 +0000 UTC m=+915.767874214" watchObservedRunningTime="2026-09-16 18:31:32.211059072 +0000 UTC m=+915.770226926"
Sep 16 18:31:43.202345 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:43.202313    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"
Sep 16 18:31:43.411995 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:43.411946    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:31:59.838989 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:31:59.838949    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:32:00.662253 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:00.662205    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"]
Sep 16 18:32:00.662568 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:00.662524    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx" podUID="baa836b3-5375-4d64-ae4e-384f4e7b21c5" containerName="payload-pre-processing" containerID="cri-o://df9becdae0b16706aae0ec8ded97ae6e4d3a71cb62ed5e4eac2727a76a47618e" gracePeriod=30
Sep 16 18:32:00.910159 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:00.910129    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"
Sep 16 18:32:01.001256 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:00.993981    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm2r7\" (UniqueName: \"kubernetes.io/projected/baa836b3-5375-4d64-ae4e-384f4e7b21c5-kube-api-access-pm2r7\") pod \"baa836b3-5375-4d64-ae4e-384f4e7b21c5\" (UID: \"baa836b3-5375-4d64-ae4e-384f4e7b21c5\") "
Sep 16 18:32:01.001256 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:00.994558    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa836b3-5375-4d64-ae4e-384f4e7b21c5-plugins-config-volume\") pod \"baa836b3-5375-4d64-ae4e-384f4e7b21c5\" (UID: \"baa836b3-5375-4d64-ae4e-384f4e7b21c5\") "
Sep 16 18:32:01.001256 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:00.995265    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa836b3-5375-4d64-ae4e-384f4e7b21c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "baa836b3-5375-4d64-ae4e-384f4e7b21c5" (UID: "baa836b3-5375-4d64-ae4e-384f4e7b21c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:32:01.006067 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:01.005914    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa836b3-5375-4d64-ae4e-384f4e7b21c5-kube-api-access-pm2r7" (OuterVolumeSpecName: "kube-api-access-pm2r7") pod "baa836b3-5375-4d64-ae4e-384f4e7b21c5" (UID: "baa836b3-5375-4d64-ae4e-384f4e7b21c5"). InnerVolumeSpecName "kube-api-access-pm2r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:32:01.096642 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:01.096600    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa836b3-5375-4d64-ae4e-384f4e7b21c5-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:32:01.096642 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:01.096643    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm2r7\" (UniqueName: \"kubernetes.io/projected/baa836b3-5375-4d64-ae4e-384f4e7b21c5-kube-api-access-pm2r7\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:32:01.374846 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:01.374753    2554 generic.go:356] "Generic (PLEG): container finished" podID="baa836b3-5375-4d64-ae4e-384f4e7b21c5" containerID="df9becdae0b16706aae0ec8ded97ae6e4d3a71cb62ed5e4eac2727a76a47618e" exitCode=0
Sep 16 18:32:01.374846 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:01.374798    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"
Sep 16 18:32:01.374846 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:01.374833    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx" event={"ID":"baa836b3-5375-4d64-ae4e-384f4e7b21c5","Type":"ContainerDied","Data":"df9becdae0b16706aae0ec8ded97ae6e4d3a71cb62ed5e4eac2727a76a47618e"}
Sep 16 18:32:01.375068 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:01.374888    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx" event={"ID":"baa836b3-5375-4d64-ae4e-384f4e7b21c5","Type":"ContainerDied","Data":"a32b705ed3740ccd52473e3e6e071f36b62597b417b8b4a51f08a842447b7bdd"}
Sep 16 18:32:01.375068 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:01.374918    2554 scope.go:117] "RemoveContainer" containerID="df9becdae0b16706aae0ec8ded97ae6e4d3a71cb62ed5e4eac2727a76a47618e"
Sep 16 18:32:01.387349 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:01.387326    2554 scope.go:117] "RemoveContainer" containerID="df9becdae0b16706aae0ec8ded97ae6e4d3a71cb62ed5e4eac2727a76a47618e"
Sep 16 18:32:01.387692 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:32:01.387675    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df9becdae0b16706aae0ec8ded97ae6e4d3a71cb62ed5e4eac2727a76a47618e\": container with ID starting with df9becdae0b16706aae0ec8ded97ae6e4d3a71cb62ed5e4eac2727a76a47618e not found: ID does not exist" containerID="df9becdae0b16706aae0ec8ded97ae6e4d3a71cb62ed5e4eac2727a76a47618e"
Sep 16 18:32:01.387751 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:01.387700    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df9becdae0b16706aae0ec8ded97ae6e4d3a71cb62ed5e4eac2727a76a47618e"} err="failed to get container status \"df9becdae0b16706aae0ec8ded97ae6e4d3a71cb62ed5e4eac2727a76a47618e\": rpc error: code = NotFound desc = could not find container \"df9becdae0b16706aae0ec8ded97ae6e4d3a71cb62ed5e4eac2727a76a47618e\": container with ID starting with df9becdae0b16706aae0ec8ded97ae6e4d3a71cb62ed5e4eac2727a76a47618e not found: ID does not exist"
Sep 16 18:32:01.407368 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:01.407327    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"]
Sep 16 18:32:01.410736 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:01.410707    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78aa4077-55d5667754-k4xqx"]
Sep 16 18:32:02.883695 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:02.883661    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baa836b3-5375-4d64-ae4e-384f4e7b21c5" path="/var/lib/kubelet/pods/baa836b3-5375-4d64-ae4e-384f4e7b21c5/volumes"
Sep 16 18:32:06.972915 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:06.972872    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"]
Sep 16 18:32:06.973389 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:06.973374    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baa836b3-5375-4d64-ae4e-384f4e7b21c5" containerName="payload-pre-processing"
Sep 16 18:32:06.973440 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:06.973391    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="baa836b3-5375-4d64-ae4e-384f4e7b21c5" containerName="payload-pre-processing"
Sep 16 18:32:06.973488 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:06.973478    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="baa836b3-5375-4d64-ae4e-384f4e7b21c5" containerName="payload-pre-processing"
Sep 16 18:32:06.976466 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:06.976441    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"
Sep 16 18:32:06.978773 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:06.978746    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-69621cf4\""
Sep 16 18:32:06.978909 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:06.978844    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-69621cf4-dockercfg-km9lm\""
Sep 16 18:32:06.988865 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:06.988830    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"]
Sep 16 18:32:06.992011 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:06.991981    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"]
Sep 16 18:32:06.994628 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:06.994605    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"
Sep 16 18:32:07.009473 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.009444    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"]
Sep 16 18:32:07.060026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.059990    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zxnb\" (UniqueName: \"kubernetes.io/projected/162975c6-45a2-4f1e-a304-af82ffaa7dc5-kube-api-access-9zxnb\") pod \"payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897\" (UID: \"162975c6-45a2-4f1e-a304-af82ffaa7dc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"
Sep 16 18:32:07.060026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.060031    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162975c6-45a2-4f1e-a304-af82ffaa7dc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897\" (UID: \"162975c6-45a2-4f1e-a304-af82ffaa7dc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"
Sep 16 18:32:07.161528 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.161485    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9754e467-7dc8-4683-a60a-1a94f93eb897-plugins-config-volume\") pod \"payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc\" (UID: \"9754e467-7dc8-4683-a60a-1a94f93eb897\") " pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"
Sep 16 18:32:07.161709 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.161571    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zxnb\" (UniqueName: \"kubernetes.io/projected/162975c6-45a2-4f1e-a304-af82ffaa7dc5-kube-api-access-9zxnb\") pod \"payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897\" (UID: \"162975c6-45a2-4f1e-a304-af82ffaa7dc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"
Sep 16 18:32:07.161709 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.161597    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162975c6-45a2-4f1e-a304-af82ffaa7dc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897\" (UID: \"162975c6-45a2-4f1e-a304-af82ffaa7dc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"
Sep 16 18:32:07.161709 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.161617    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw5fk\" (UniqueName: \"kubernetes.io/projected/9754e467-7dc8-4683-a60a-1a94f93eb897-kube-api-access-vw5fk\") pod \"payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc\" (UID: \"9754e467-7dc8-4683-a60a-1a94f93eb897\") " pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"
Sep 16 18:32:07.162403 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.162382    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162975c6-45a2-4f1e-a304-af82ffaa7dc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897\" (UID: \"162975c6-45a2-4f1e-a304-af82ffaa7dc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"
Sep 16 18:32:07.186220 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.186182    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zxnb\" (UniqueName: \"kubernetes.io/projected/162975c6-45a2-4f1e-a304-af82ffaa7dc5-kube-api-access-9zxnb\") pod \"payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897\" (UID: \"162975c6-45a2-4f1e-a304-af82ffaa7dc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"
Sep 16 18:32:07.262950 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.262906    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9754e467-7dc8-4683-a60a-1a94f93eb897-plugins-config-volume\") pod \"payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc\" (UID: \"9754e467-7dc8-4683-a60a-1a94f93eb897\") " pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"
Sep 16 18:32:07.263154 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.262993    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw5fk\" (UniqueName: \"kubernetes.io/projected/9754e467-7dc8-4683-a60a-1a94f93eb897-kube-api-access-vw5fk\") pod \"payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc\" (UID: \"9754e467-7dc8-4683-a60a-1a94f93eb897\") " pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"
Sep 16 18:32:07.263675 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.263654    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9754e467-7dc8-4683-a60a-1a94f93eb897-plugins-config-volume\") pod \"payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc\" (UID: \"9754e467-7dc8-4683-a60a-1a94f93eb897\") " pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"
Sep 16 18:32:07.272112 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.272077    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw5fk\" (UniqueName: \"kubernetes.io/projected/9754e467-7dc8-4683-a60a-1a94f93eb897-kube-api-access-vw5fk\") pod \"payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc\" (UID: \"9754e467-7dc8-4683-a60a-1a94f93eb897\") " pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"
Sep 16 18:32:07.287934 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.287899    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"
Sep 16 18:32:07.305905 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.305872    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"
Sep 16 18:32:07.567800 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.567773    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"]
Sep 16 18:32:07.569886 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:32:07.569864    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162975c6_45a2_4f1e_a304_af82ffaa7dc5.slice/crio-ef33a9de0cd1e8411e6bb1af7400b06405df7d2f28b0b754c291d21ecaa84ce7 WatchSource:0}: Error finding container ef33a9de0cd1e8411e6bb1af7400b06405df7d2f28b0b754c291d21ecaa84ce7: Status 404 returned error can't find the container with id ef33a9de0cd1e8411e6bb1af7400b06405df7d2f28b0b754c291d21ecaa84ce7
Sep 16 18:32:07.592258 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:07.592202    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"]
Sep 16 18:32:07.603414 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:32:07.603385    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9754e467_7dc8_4683_a60a_1a94f93eb897.slice/crio-e10d493de2ba3c22430b917bc09fc1045a558db2e1f81c910775fad8863e6ddc WatchSource:0}: Error finding container e10d493de2ba3c22430b917bc09fc1045a558db2e1f81c910775fad8863e6ddc: Status 404 returned error can't find the container with id e10d493de2ba3c22430b917bc09fc1045a558db2e1f81c910775fad8863e6ddc
Sep 16 18:32:08.416874 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:08.416818    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc" event={"ID":"9754e467-7dc8-4683-a60a-1a94f93eb897","Type":"ContainerStarted","Data":"6d409276e6c312011790762044d4a5b99dec99fed4f04019883ed86e9ee6b37d"}
Sep 16 18:32:08.417356 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:08.416977    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"
Sep 16 18:32:08.417356 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:08.417044    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc" event={"ID":"9754e467-7dc8-4683-a60a-1a94f93eb897","Type":"ContainerStarted","Data":"e10d493de2ba3c22430b917bc09fc1045a558db2e1f81c910775fad8863e6ddc"}
Sep 16 18:32:08.418452 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:08.418423    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897" event={"ID":"162975c6-45a2-4f1e-a304-af82ffaa7dc5","Type":"ContainerStarted","Data":"9b9a210f9d227b4796ef9703273c42150df26d627664b3e6d88ac80dc8ec571f"}
Sep 16 18:32:08.418584 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:08.418467    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897" event={"ID":"162975c6-45a2-4f1e-a304-af82ffaa7dc5","Type":"ContainerStarted","Data":"ef33a9de0cd1e8411e6bb1af7400b06405df7d2f28b0b754c291d21ecaa84ce7"}
Sep 16 18:32:08.418584 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:08.418513    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"
Sep 16 18:32:08.435118 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:08.435058    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc" podStartSLOduration=2.435039134 podStartE2EDuration="2.435039134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:32:08.432740706 +0000 UTC m=+951.991908539" watchObservedRunningTime="2026-09-16 18:32:08.435039134 +0000 UTC m=+951.994207027"
Sep 16 18:32:08.449399 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:08.449337    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897" podStartSLOduration=2.449320222 podStartE2EDuration="2.449320222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:32:08.447994765 +0000 UTC m=+952.007162600" watchObservedRunningTime="2026-09-16 18:32:08.449320222 +0000 UTC m=+952.008488087"
Sep 16 18:32:19.426318 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:19.426275    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"
Sep 16 18:32:19.426692 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:19.426349    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"
Sep 16 18:32:19.921965 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:19.921925    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:32:34.557866 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:34.557829    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:32:41.323088 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.323008    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"]
Sep 16 18:32:41.323510 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.323249    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc" podUID="9754e467-7dc8-4683-a60a-1a94f93eb897" containerName="payload-processing" containerID="cri-o://6d409276e6c312011790762044d4a5b99dec99fed4f04019883ed86e9ee6b37d" gracePeriod=30
Sep 16 18:32:41.338408 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.338378    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"]
Sep 16 18:32:41.338610 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.338587    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897" podUID="162975c6-45a2-4f1e-a304-af82ffaa7dc5" containerName="payload-pre-processing" containerID="cri-o://9b9a210f9d227b4796ef9703273c42150df26d627664b3e6d88ac80dc8ec571f" gracePeriod=30
Sep 16 18:32:41.596323 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.596283    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"
Sep 16 18:32:41.617206 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.616786    2554 generic.go:356] "Generic (PLEG): container finished" podID="9754e467-7dc8-4683-a60a-1a94f93eb897" containerID="6d409276e6c312011790762044d4a5b99dec99fed4f04019883ed86e9ee6b37d" exitCode=0
Sep 16 18:32:41.617206 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.616911    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc" event={"ID":"9754e467-7dc8-4683-a60a-1a94f93eb897","Type":"ContainerDied","Data":"6d409276e6c312011790762044d4a5b99dec99fed4f04019883ed86e9ee6b37d"}
Sep 16 18:32:41.617206 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.616977    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc" event={"ID":"9754e467-7dc8-4683-a60a-1a94f93eb897","Type":"ContainerDied","Data":"e10d493de2ba3c22430b917bc09fc1045a558db2e1f81c910775fad8863e6ddc"}
Sep 16 18:32:41.617206 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.617013    2554 scope.go:117] "RemoveContainer" containerID="6d409276e6c312011790762044d4a5b99dec99fed4f04019883ed86e9ee6b37d"
Sep 16 18:32:41.617725 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.617697    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"
Sep 16 18:32:41.619858 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.619679    2554 generic.go:356] "Generic (PLEG): container finished" podID="162975c6-45a2-4f1e-a304-af82ffaa7dc5" containerID="9b9a210f9d227b4796ef9703273c42150df26d627664b3e6d88ac80dc8ec571f" exitCode=0
Sep 16 18:32:41.619964 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.619867    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897" event={"ID":"162975c6-45a2-4f1e-a304-af82ffaa7dc5","Type":"ContainerDied","Data":"9b9a210f9d227b4796ef9703273c42150df26d627664b3e6d88ac80dc8ec571f"}
Sep 16 18:32:41.619964 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.619915    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897" event={"ID":"162975c6-45a2-4f1e-a304-af82ffaa7dc5","Type":"ContainerDied","Data":"ef33a9de0cd1e8411e6bb1af7400b06405df7d2f28b0b754c291d21ecaa84ce7"}
Sep 16 18:32:41.619964 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.619936    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef33a9de0cd1e8411e6bb1af7400b06405df7d2f28b0b754c291d21ecaa84ce7"
Sep 16 18:32:41.627409 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.627388    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"
Sep 16 18:32:41.629347 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.629327    2554 scope.go:117] "RemoveContainer" containerID="6d409276e6c312011790762044d4a5b99dec99fed4f04019883ed86e9ee6b37d"
Sep 16 18:32:41.629649 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:32:41.629616    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d409276e6c312011790762044d4a5b99dec99fed4f04019883ed86e9ee6b37d\": container with ID starting with 6d409276e6c312011790762044d4a5b99dec99fed4f04019883ed86e9ee6b37d not found: ID does not exist" containerID="6d409276e6c312011790762044d4a5b99dec99fed4f04019883ed86e9ee6b37d"
Sep 16 18:32:41.629735 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.629655    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d409276e6c312011790762044d4a5b99dec99fed4f04019883ed86e9ee6b37d"} err="failed to get container status \"6d409276e6c312011790762044d4a5b99dec99fed4f04019883ed86e9ee6b37d\": rpc error: code = NotFound desc = could not find container \"6d409276e6c312011790762044d4a5b99dec99fed4f04019883ed86e9ee6b37d\": container with ID starting with 6d409276e6c312011790762044d4a5b99dec99fed4f04019883ed86e9ee6b37d not found: ID does not exist"
Sep 16 18:32:41.731668 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.731630    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw5fk\" (UniqueName: \"kubernetes.io/projected/9754e467-7dc8-4683-a60a-1a94f93eb897-kube-api-access-vw5fk\") pod \"9754e467-7dc8-4683-a60a-1a94f93eb897\" (UID: \"9754e467-7dc8-4683-a60a-1a94f93eb897\") "
Sep 16 18:32:41.731877 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.731679    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162975c6-45a2-4f1e-a304-af82ffaa7dc5-plugins-config-volume\") pod \"162975c6-45a2-4f1e-a304-af82ffaa7dc5\" (UID: \"162975c6-45a2-4f1e-a304-af82ffaa7dc5\") "
Sep 16 18:32:41.731877 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.731753    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zxnb\" (UniqueName: \"kubernetes.io/projected/162975c6-45a2-4f1e-a304-af82ffaa7dc5-kube-api-access-9zxnb\") pod \"162975c6-45a2-4f1e-a304-af82ffaa7dc5\" (UID: \"162975c6-45a2-4f1e-a304-af82ffaa7dc5\") "
Sep 16 18:32:41.731877 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.731772    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9754e467-7dc8-4683-a60a-1a94f93eb897-plugins-config-volume\") pod \"9754e467-7dc8-4683-a60a-1a94f93eb897\" (UID: \"9754e467-7dc8-4683-a60a-1a94f93eb897\") "
Sep 16 18:32:41.732095 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.732066    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/162975c6-45a2-4f1e-a304-af82ffaa7dc5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "162975c6-45a2-4f1e-a304-af82ffaa7dc5" (UID: "162975c6-45a2-4f1e-a304-af82ffaa7dc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:32:41.732163 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.732143    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9754e467-7dc8-4683-a60a-1a94f93eb897-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9754e467-7dc8-4683-a60a-1a94f93eb897" (UID: "9754e467-7dc8-4683-a60a-1a94f93eb897"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:32:41.733858 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.733837    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162975c6-45a2-4f1e-a304-af82ffaa7dc5-kube-api-access-9zxnb" (OuterVolumeSpecName: "kube-api-access-9zxnb") pod "162975c6-45a2-4f1e-a304-af82ffaa7dc5" (UID: "162975c6-45a2-4f1e-a304-af82ffaa7dc5"). InnerVolumeSpecName "kube-api-access-9zxnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:32:41.733932 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.733892    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9754e467-7dc8-4683-a60a-1a94f93eb897-kube-api-access-vw5fk" (OuterVolumeSpecName: "kube-api-access-vw5fk") pod "9754e467-7dc8-4683-a60a-1a94f93eb897" (UID: "9754e467-7dc8-4683-a60a-1a94f93eb897"). InnerVolumeSpecName "kube-api-access-vw5fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:32:41.833127 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.833030    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw5fk\" (UniqueName: \"kubernetes.io/projected/9754e467-7dc8-4683-a60a-1a94f93eb897-kube-api-access-vw5fk\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:32:41.833127 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.833062    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162975c6-45a2-4f1e-a304-af82ffaa7dc5-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:32:41.833127 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.833072    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zxnb\" (UniqueName: \"kubernetes.io/projected/162975c6-45a2-4f1e-a304-af82ffaa7dc5-kube-api-access-9zxnb\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:32:41.833127 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.833083    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9754e467-7dc8-4683-a60a-1a94f93eb897-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:32:41.947315 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.947278    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"]
Sep 16 18:32:41.954578 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:41.954542    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-69621cf4-54db8b9477-m9cgc"]
Sep 16 18:32:42.626634 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:42.626553    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"
Sep 16 18:32:42.653963 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:42.653927    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"]
Sep 16 18:32:42.661642 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:42.661613    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-69621cf4-5d7dfc446b-8d897"]
Sep 16 18:32:42.885095 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:42.885010    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="162975c6-45a2-4f1e-a304-af82ffaa7dc5" path="/var/lib/kubelet/pods/162975c6-45a2-4f1e-a304-af82ffaa7dc5/volumes"
Sep 16 18:32:42.885441 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:42.885425    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9754e467-7dc8-4683-a60a-1a94f93eb897" path="/var/lib/kubelet/pods/9754e467-7dc8-4683-a60a-1a94f93eb897/volumes"
Sep 16 18:32:43.607210 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.607173    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"]
Sep 16 18:32:43.607663 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.607648    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="162975c6-45a2-4f1e-a304-af82ffaa7dc5" containerName="payload-pre-processing"
Sep 16 18:32:43.607708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.607665    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="162975c6-45a2-4f1e-a304-af82ffaa7dc5" containerName="payload-pre-processing"
Sep 16 18:32:43.607708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.607680    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9754e467-7dc8-4683-a60a-1a94f93eb897" containerName="payload-processing"
Sep 16 18:32:43.607708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.607687    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9754e467-7dc8-4683-a60a-1a94f93eb897" containerName="payload-processing"
Sep 16 18:32:43.607807 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.607756    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="162975c6-45a2-4f1e-a304-af82ffaa7dc5" containerName="payload-pre-processing"
Sep 16 18:32:43.607807 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.607766    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9754e467-7dc8-4683-a60a-1a94f93eb897" containerName="payload-processing"
Sep 16 18:32:43.611030 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.611010    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.614808 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.614789    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-7e8b5de5-openshift-default-dockercfg-rwts9\""
Sep 16 18:32:43.627995 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.627970    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"]
Sep 16 18:32:43.753592 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.753544    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-podinfo\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.753592 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.753593    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-data\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.753804 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.753663    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-envoy\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.753804 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.753709    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-workload-socket\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.753804 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.753746    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkzw4\" (UniqueName: \"kubernetes.io/projected/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-kube-api-access-kkzw4\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.753804 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.753794    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-credential-socket\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.753979 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.753840    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istiod-ca-cert\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.753979 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.753865    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-workload-certs\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.753979 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.753947    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-token\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.854637 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.854591    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-workload-certs\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.854814 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.854691    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-token\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.854814 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.854752    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-podinfo\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.854814 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.854788    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-data\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.854970 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.854827    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-envoy\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.854970 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.854868    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-workload-socket\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.854970 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.854914    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkzw4\" (UniqueName: \"kubernetes.io/projected/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-kube-api-access-kkzw4\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.854970 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.854944    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-credential-socket\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.855171 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.854984    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istiod-ca-cert\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.855261 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.855165    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-data\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.855329 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.855318    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-workload-certs\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.855379 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.855342    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-workload-socket\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.855379 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.855357    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-credential-socket\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.856043 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.856015    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istiod-ca-cert\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.857422 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.857355    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-envoy\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.857519 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.857507    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-podinfo\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.875825 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.875789    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-token\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.875966 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.875857    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkzw4\" (UniqueName: \"kubernetes.io/projected/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-kube-api-access-kkzw4\") pod \"e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") " pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:43.920909 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:43.920863    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:44.214891 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:44.214865    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"]
Sep 16 18:32:44.216936 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:32:44.216902    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c849fc_98e8_46f1_b0e3_ea1a07ff4421.slice/crio-40e250c7022bca41948a0e2956c50eb717bd3437db5bb7b35d6c6af79cc40fc6 WatchSource:0}: Error finding container 40e250c7022bca41948a0e2956c50eb717bd3437db5bb7b35d6c6af79cc40fc6: Status 404 returned error can't find the container with id 40e250c7022bca41948a0e2956c50eb717bd3437db5bb7b35d6c6af79cc40fc6
Sep 16 18:32:44.219157 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:44.219125    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:32:44.219246 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:44.219191    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:32:44.219296 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:44.219221    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:32:44.640312 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:44.640268    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r" event={"ID":"08c849fc-98e8-46f1-b0e3-ea1a07ff4421","Type":"ContainerStarted","Data":"b2c33528dfaa147a833781e81442f7502691fe711abe4d8e079d6c89389e8b3f"}
Sep 16 18:32:44.640817 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:44.640362    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r" event={"ID":"08c849fc-98e8-46f1-b0e3-ea1a07ff4421","Type":"ContainerStarted","Data":"40e250c7022bca41948a0e2956c50eb717bd3437db5bb7b35d6c6af79cc40fc6"}
Sep 16 18:32:44.677318 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:44.677258    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r" podStartSLOduration=1.677227719 podStartE2EDuration="1.677227719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:32:44.675885856 +0000 UTC m=+988.235053690" watchObservedRunningTime="2026-09-16 18:32:44.677227719 +0000 UTC m=+988.236395552"
Sep 16 18:32:44.921170 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:44.921080    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:45.926481 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:45.926445    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:46.654559 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:46.654528    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:46.655499 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:46.655479    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:32:49.183748 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:49.183711    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"]
Sep 16 18:32:49.187396 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:49.187373    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"
Sep 16 18:32:49.190696 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:49.190674    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-7e8b5de5\""
Sep 16 18:32:49.190828 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:49.190701    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-7e8b5de5-dockercfg-rzcq8\""
Sep 16 18:32:49.204372 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:49.204342    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"]
Sep 16 18:32:49.213691 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:49.213661    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c74e09-3a8d-4ba7-abfd-92e607c43d92-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7\" (UID: \"19c74e09-3a8d-4ba7-abfd-92e607c43d92\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"
Sep 16 18:32:49.213861 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:49.213707    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxrv9\" (UniqueName: \"kubernetes.io/projected/19c74e09-3a8d-4ba7-abfd-92e607c43d92-kube-api-access-rxrv9\") pod \"payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7\" (UID: \"19c74e09-3a8d-4ba7-abfd-92e607c43d92\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"
Sep 16 18:32:49.314305 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:49.314262    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c74e09-3a8d-4ba7-abfd-92e607c43d92-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7\" (UID: \"19c74e09-3a8d-4ba7-abfd-92e607c43d92\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"
Sep 16 18:32:49.314305 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:49.314312    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxrv9\" (UniqueName: \"kubernetes.io/projected/19c74e09-3a8d-4ba7-abfd-92e607c43d92-kube-api-access-rxrv9\") pod \"payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7\" (UID: \"19c74e09-3a8d-4ba7-abfd-92e607c43d92\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"
Sep 16 18:32:49.315298 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:49.315271    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c74e09-3a8d-4ba7-abfd-92e607c43d92-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7\" (UID: \"19c74e09-3a8d-4ba7-abfd-92e607c43d92\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"
Sep 16 18:32:49.333154 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:49.326090    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxrv9\" (UniqueName: \"kubernetes.io/projected/19c74e09-3a8d-4ba7-abfd-92e607c43d92-kube-api-access-rxrv9\") pod \"payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7\" (UID: \"19c74e09-3a8d-4ba7-abfd-92e607c43d92\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"
Sep 16 18:32:49.499022 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:49.498972    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"
Sep 16 18:32:49.781642 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:49.781612    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"]
Sep 16 18:32:49.784189 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:32:49.784149    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c74e09_3a8d_4ba7_abfd_92e607c43d92.slice/crio-243bc833a10f6bcd27a963f74e4b3f4e8348c575e12e1545e5aa7c5f64e68db0 WatchSource:0}: Error finding container 243bc833a10f6bcd27a963f74e4b3f4e8348c575e12e1545e5aa7c5f64e68db0: Status 404 returned error can't find the container with id 243bc833a10f6bcd27a963f74e4b3f4e8348c575e12e1545e5aa7c5f64e68db0
Sep 16 18:32:50.679313 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:50.679274    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7" event={"ID":"19c74e09-3a8d-4ba7-abfd-92e607c43d92","Type":"ContainerStarted","Data":"01011cafeb4fcc2a8f1586c75d52599605e20a59fd4bca9697e387b0bfa492fe"}
Sep 16 18:32:50.679730 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:50.679330    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7" event={"ID":"19c74e09-3a8d-4ba7-abfd-92e607c43d92","Type":"ContainerStarted","Data":"243bc833a10f6bcd27a963f74e4b3f4e8348c575e12e1545e5aa7c5f64e68db0"}
Sep 16 18:32:50.679730 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:50.679365    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"
Sep 16 18:32:50.704430 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:50.704373    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7" podStartSLOduration=1.7043529720000001 podStartE2EDuration="1.704352972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:32:50.702309147 +0000 UTC m=+994.261476982" watchObservedRunningTime="2026-09-16 18:32:50.704352972 +0000 UTC m=+994.263520805"
Sep 16 18:32:54.638248 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:54.638202    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:32:59.979277 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:59.979075    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"]
Sep 16 18:32:59.979770 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:32:59.979357    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq" podUID="627dd71a-bccc-4627-b52f-6914d4060a43" containerName="istio-proxy" containerID="cri-o://1ee059c5d7867b86f5d13d9c252ce7e18661829e9f86bf1ab63cff6d9bed3da1" gracePeriod=30
Sep 16 18:33:00.668803 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:00.668760    2554 patch_prober.go:28] interesting pod/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 18:33:00.668999 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:00.668856    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq" podUID="627dd71a-bccc-4627-b52f-6914d4060a43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:33:01.687182 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:01.687150    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"
Sep 16 18:33:05.256178 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.256097    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:33:05.392257 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.392198    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-workload-certs\") pod \"627dd71a-bccc-4627-b52f-6914d4060a43\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") "
Sep 16 18:33:05.392464 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.392297    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-istio-envoy\") pod \"627dd71a-bccc-4627-b52f-6914d4060a43\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") "
Sep 16 18:33:05.392464 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.392351    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627dd71a-bccc-4627-b52f-6914d4060a43-istio-token\") pod \"627dd71a-bccc-4627-b52f-6914d4060a43\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") "
Sep 16 18:33:05.392464 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.392409    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-credential-socket\") pod \"627dd71a-bccc-4627-b52f-6914d4060a43\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") "
Sep 16 18:33:05.392464 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.392449    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-istio-data\") pod \"627dd71a-bccc-4627-b52f-6914d4060a43\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") "
Sep 16 18:33:05.392731 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.392489    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7mpn\" (UniqueName: \"kubernetes.io/projected/627dd71a-bccc-4627-b52f-6914d4060a43-kube-api-access-s7mpn\") pod \"627dd71a-bccc-4627-b52f-6914d4060a43\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") "
Sep 16 18:33:05.392731 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.392546    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627dd71a-bccc-4627-b52f-6914d4060a43-istiod-ca-cert\") pod \"627dd71a-bccc-4627-b52f-6914d4060a43\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") "
Sep 16 18:33:05.392731 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.392556    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "627dd71a-bccc-4627-b52f-6914d4060a43" (UID: "627dd71a-bccc-4627-b52f-6914d4060a43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:33:05.392731 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.392589    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-workload-socket\") pod \"627dd71a-bccc-4627-b52f-6914d4060a43\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") "
Sep 16 18:33:05.392731 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.392657    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627dd71a-bccc-4627-b52f-6914d4060a43-istio-podinfo\") pod \"627dd71a-bccc-4627-b52f-6914d4060a43\" (UID: \"627dd71a-bccc-4627-b52f-6914d4060a43\") "
Sep 16 18:33:05.393005 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.392983    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-istio-data" (OuterVolumeSpecName: "istio-data") pod "627dd71a-bccc-4627-b52f-6914d4060a43" (UID: "627dd71a-bccc-4627-b52f-6914d4060a43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:33:05.393064 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.393000    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "627dd71a-bccc-4627-b52f-6914d4060a43" (UID: "627dd71a-bccc-4627-b52f-6914d4060a43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:33:05.393064 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.393028    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:05.393166 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.393073    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/627dd71a-bccc-4627-b52f-6914d4060a43-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "627dd71a-bccc-4627-b52f-6914d4060a43" (UID: "627dd71a-bccc-4627-b52f-6914d4060a43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:33:05.393245 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.393161    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "627dd71a-bccc-4627-b52f-6914d4060a43" (UID: "627dd71a-bccc-4627-b52f-6914d4060a43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:33:05.395209 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.395154    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627dd71a-bccc-4627-b52f-6914d4060a43-istio-token" (OuterVolumeSpecName: "istio-token") pod "627dd71a-bccc-4627-b52f-6914d4060a43" (UID: "627dd71a-bccc-4627-b52f-6914d4060a43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:33:05.395209 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.395173    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627dd71a-bccc-4627-b52f-6914d4060a43-kube-api-access-s7mpn" (OuterVolumeSpecName: "kube-api-access-s7mpn") pod "627dd71a-bccc-4627-b52f-6914d4060a43" (UID: "627dd71a-bccc-4627-b52f-6914d4060a43"). InnerVolumeSpecName "kube-api-access-s7mpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:33:05.395462 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.395445    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/627dd71a-bccc-4627-b52f-6914d4060a43-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "627dd71a-bccc-4627-b52f-6914d4060a43" (UID: "627dd71a-bccc-4627-b52f-6914d4060a43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:33:05.395611 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.395597    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "627dd71a-bccc-4627-b52f-6914d4060a43" (UID: "627dd71a-bccc-4627-b52f-6914d4060a43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:33:05.494084 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.494050    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:05.494084 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.494081    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627dd71a-bccc-4627-b52f-6914d4060a43-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:05.494084 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.494092    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:05.495210 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.494100    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:05.495210 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.494109    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7mpn\" (UniqueName: \"kubernetes.io/projected/627dd71a-bccc-4627-b52f-6914d4060a43-kube-api-access-s7mpn\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:05.495210 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.494119    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627dd71a-bccc-4627-b52f-6914d4060a43-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:05.495210 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.494128    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627dd71a-bccc-4627-b52f-6914d4060a43-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:05.495210 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.494135    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627dd71a-bccc-4627-b52f-6914d4060a43-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:05.785596 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.785503    2554 generic.go:356] "Generic (PLEG): container finished" podID="627dd71a-bccc-4627-b52f-6914d4060a43" containerID="1ee059c5d7867b86f5d13d9c252ce7e18661829e9f86bf1ab63cff6d9bed3da1" exitCode=0
Sep 16 18:33:05.785596 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.785550    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"
Sep 16 18:33:05.785596 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.785583    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq" event={"ID":"627dd71a-bccc-4627-b52f-6914d4060a43","Type":"ContainerDied","Data":"1ee059c5d7867b86f5d13d9c252ce7e18661829e9f86bf1ab63cff6d9bed3da1"}
Sep 16 18:33:05.785849 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.785639    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq" event={"ID":"627dd71a-bccc-4627-b52f-6914d4060a43","Type":"ContainerDied","Data":"6bc3a064c1f9cc7f9e6540df2e6d7c5d475e9d61640114afc29f287de351effb"}
Sep 16 18:33:05.785849 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.785666    2554 scope.go:117] "RemoveContainer" containerID="1ee059c5d7867b86f5d13d9c252ce7e18661829e9f86bf1ab63cff6d9bed3da1"
Sep 16 18:33:05.787474 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.787449    2554 generic.go:356] "Generic (PLEG): container finished" podID="08c849fc-98e8-46f1-b0e3-ea1a07ff4421" containerID="b2c33528dfaa147a833781e81442f7502691fe711abe4d8e079d6c89389e8b3f" exitCode=0
Sep 16 18:33:05.787584 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.787477    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r" event={"ID":"08c849fc-98e8-46f1-b0e3-ea1a07ff4421","Type":"ContainerDied","Data":"b2c33528dfaa147a833781e81442f7502691fe711abe4d8e079d6c89389e8b3f"}
Sep 16 18:33:05.787886 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.787872    2554 scope.go:117] "RemoveContainer" containerID="b2c33528dfaa147a833781e81442f7502691fe711abe4d8e079d6c89389e8b3f"
Sep 16 18:33:05.789057 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.789027    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:33:05.789139 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.789096    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:33:05.789139 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.789133    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:33:05.796083 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.796063    2554 scope.go:117] "RemoveContainer" containerID="bf48b01c1c6beb71451e307277690f5dd85d64cfcadbc819b133defecac22ebd"
Sep 16 18:33:05.805486 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.805456    2554 scope.go:117] "RemoveContainer" containerID="1ee059c5d7867b86f5d13d9c252ce7e18661829e9f86bf1ab63cff6d9bed3da1"
Sep 16 18:33:05.805893 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:33:05.805828    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee059c5d7867b86f5d13d9c252ce7e18661829e9f86bf1ab63cff6d9bed3da1\": container with ID starting with 1ee059c5d7867b86f5d13d9c252ce7e18661829e9f86bf1ab63cff6d9bed3da1 not found: ID does not exist" containerID="1ee059c5d7867b86f5d13d9c252ce7e18661829e9f86bf1ab63cff6d9bed3da1"
Sep 16 18:33:05.805893 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.805872    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee059c5d7867b86f5d13d9c252ce7e18661829e9f86bf1ab63cff6d9bed3da1"} err="failed to get container status \"1ee059c5d7867b86f5d13d9c252ce7e18661829e9f86bf1ab63cff6d9bed3da1\": rpc error: code = NotFound desc = could not find container \"1ee059c5d7867b86f5d13d9c252ce7e18661829e9f86bf1ab63cff6d9bed3da1\": container with ID starting with 1ee059c5d7867b86f5d13d9c252ce7e18661829e9f86bf1ab63cff6d9bed3da1 not found: ID does not exist"
Sep 16 18:33:05.806018 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.805897    2554 scope.go:117] "RemoveContainer" containerID="bf48b01c1c6beb71451e307277690f5dd85d64cfcadbc819b133defecac22ebd"
Sep 16 18:33:05.806170 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:33:05.806152    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf48b01c1c6beb71451e307277690f5dd85d64cfcadbc819b133defecac22ebd\": container with ID starting with bf48b01c1c6beb71451e307277690f5dd85d64cfcadbc819b133defecac22ebd not found: ID does not exist" containerID="bf48b01c1c6beb71451e307277690f5dd85d64cfcadbc819b133defecac22ebd"
Sep 16 18:33:05.806225 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.806177    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf48b01c1c6beb71451e307277690f5dd85d64cfcadbc819b133defecac22ebd"} err="failed to get container status \"bf48b01c1c6beb71451e307277690f5dd85d64cfcadbc819b133defecac22ebd\": rpc error: code = NotFound desc = could not find container \"bf48b01c1c6beb71451e307277690f5dd85d64cfcadbc819b133defecac22ebd\": container with ID starting with bf48b01c1c6beb71451e307277690f5dd85d64cfcadbc819b133defecac22ebd not found: ID does not exist"
Sep 16 18:33:05.850678 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.850654    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"]
Sep 16 18:33:05.861656 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:05.860895    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-305998-openshift-default-cc84f95f8-wxtqq"]
Sep 16 18:33:06.796121 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:06.796083    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r" event={"ID":"08c849fc-98e8-46f1-b0e3-ea1a07ff4421","Type":"ContainerStarted","Data":"0cebbcde18dcc9b825701372328b3f1c4ae7fc1b15b4d202f16c20464bccd102"}
Sep 16 18:33:06.886700 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:06.886668    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="627dd71a-bccc-4627-b52f-6914d4060a43" path="/var/lib/kubelet/pods/627dd71a-bccc-4627-b52f-6914d4060a43/volumes"
Sep 16 18:33:06.921205 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:06.921168    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:33:06.921533 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:06.921510    2554 patch_prober.go:28] interesting pod/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 16 18:33:06.921623 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:06.921566    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r" podUID="08c849fc-98e8-46f1-b0e3-ea1a07ff4421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 16 18:33:07.559260 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.554262    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"]
Sep 16 18:33:07.559260 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.555025    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="627dd71a-bccc-4627-b52f-6914d4060a43" containerName="istio-proxy"
Sep 16 18:33:07.559260 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.555049    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="627dd71a-bccc-4627-b52f-6914d4060a43" containerName="istio-proxy"
Sep 16 18:33:07.559260 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.555083    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="627dd71a-bccc-4627-b52f-6914d4060a43" containerName="istio-proxy"
Sep 16 18:33:07.559260 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.555092    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="627dd71a-bccc-4627-b52f-6914d4060a43" containerName="istio-proxy"
Sep 16 18:33:07.559260 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.555198    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="627dd71a-bccc-4627-b52f-6914d4060a43" containerName="istio-proxy"
Sep 16 18:33:07.559260 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.555214    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="627dd71a-bccc-4627-b52f-6914d4060a43" containerName="istio-proxy"
Sep 16 18:33:07.569257 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.567539    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"
Sep 16 18:33:07.571149 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.571110    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-2c3173\""
Sep 16 18:33:07.572555 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.572529    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"]
Sep 16 18:33:07.589759 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.589729    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"]
Sep 16 18:33:07.592708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.592682    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"
Sep 16 18:33:07.595653 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.595631    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-2c3173-dockercfg-xs5st\""
Sep 16 18:33:07.607372 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.607342    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"]
Sep 16 18:33:07.715676 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.715636    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221247e9-8225-412e-856f-2efb8965e299-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn\" (UID: \"221247e9-8225-412e-856f-2efb8965e299\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"
Sep 16 18:33:07.715872 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.715683    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2qvh\" (UniqueName: \"kubernetes.io/projected/221247e9-8225-412e-856f-2efb8965e299-kube-api-access-w2qvh\") pod \"payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn\" (UID: \"221247e9-8225-412e-856f-2efb8965e299\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"
Sep 16 18:33:07.715872 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.715819    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25jcz\" (UniqueName: \"kubernetes.io/projected/817087e7-6e5c-44ab-bc24-28dd0a358bd4-kube-api-access-25jcz\") pod \"payload-processing-e2e-explicit-2c3173-5c75945489-dws9n\" (UID: \"817087e7-6e5c-44ab-bc24-28dd0a358bd4\") " pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"
Sep 16 18:33:07.715947 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.715880    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/817087e7-6e5c-44ab-bc24-28dd0a358bd4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2c3173-5c75945489-dws9n\" (UID: \"817087e7-6e5c-44ab-bc24-28dd0a358bd4\") " pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"
Sep 16 18:33:07.816960 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.816862    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221247e9-8225-412e-856f-2efb8965e299-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn\" (UID: \"221247e9-8225-412e-856f-2efb8965e299\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"
Sep 16 18:33:07.816960 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.816902    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2qvh\" (UniqueName: \"kubernetes.io/projected/221247e9-8225-412e-856f-2efb8965e299-kube-api-access-w2qvh\") pod \"payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn\" (UID: \"221247e9-8225-412e-856f-2efb8965e299\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"
Sep 16 18:33:07.816960 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.816944    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25jcz\" (UniqueName: \"kubernetes.io/projected/817087e7-6e5c-44ab-bc24-28dd0a358bd4-kube-api-access-25jcz\") pod \"payload-processing-e2e-explicit-2c3173-5c75945489-dws9n\" (UID: \"817087e7-6e5c-44ab-bc24-28dd0a358bd4\") " pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"
Sep 16 18:33:07.817484 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.816980    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/817087e7-6e5c-44ab-bc24-28dd0a358bd4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2c3173-5c75945489-dws9n\" (UID: \"817087e7-6e5c-44ab-bc24-28dd0a358bd4\") " pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"
Sep 16 18:33:07.817634 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.817612    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221247e9-8225-412e-856f-2efb8965e299-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn\" (UID: \"221247e9-8225-412e-856f-2efb8965e299\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"
Sep 16 18:33:07.817732 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.817713    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/817087e7-6e5c-44ab-bc24-28dd0a358bd4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2c3173-5c75945489-dws9n\" (UID: \"817087e7-6e5c-44ab-bc24-28dd0a358bd4\") " pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"
Sep 16 18:33:07.835513 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.835481    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25jcz\" (UniqueName: \"kubernetes.io/projected/817087e7-6e5c-44ab-bc24-28dd0a358bd4-kube-api-access-25jcz\") pod \"payload-processing-e2e-explicit-2c3173-5c75945489-dws9n\" (UID: \"817087e7-6e5c-44ab-bc24-28dd0a358bd4\") " pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"
Sep 16 18:33:07.835513 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.835507    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2qvh\" (UniqueName: \"kubernetes.io/projected/221247e9-8225-412e-856f-2efb8965e299-kube-api-access-w2qvh\") pod \"payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn\" (UID: \"221247e9-8225-412e-856f-2efb8965e299\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"
Sep 16 18:33:07.883143 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.883105    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"
Sep 16 18:33:07.904282 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.904253    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"
Sep 16 18:33:07.921964 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.921926    2554 patch_prober.go:28] interesting pod/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 16 18:33:07.922136 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:07.921989    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r" podUID="08c849fc-98e8-46f1-b0e3-ea1a07ff4421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 16 18:33:08.194114 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:08.194086    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"]
Sep 16 18:33:08.195738 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:33:08.195709    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817087e7_6e5c_44ab_bc24_28dd0a358bd4.slice/crio-cf3fb02f6ecdc566a2573a6b3c31ce4d2388ef45111c4607c7df3be6d91db767 WatchSource:0}: Error finding container cf3fb02f6ecdc566a2573a6b3c31ce4d2388ef45111c4607c7df3be6d91db767: Status 404 returned error can't find the container with id cf3fb02f6ecdc566a2573a6b3c31ce4d2388ef45111c4607c7df3be6d91db767
Sep 16 18:33:08.235101 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:08.235061    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"]
Sep 16 18:33:08.808992 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:08.808951    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn" event={"ID":"221247e9-8225-412e-856f-2efb8965e299","Type":"ContainerStarted","Data":"b38f8c098109a1b28e96d76f2b88bd7ee884306af362b5590ca2498e9b85186f"}
Sep 16 18:33:08.809202 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:08.809022    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"
Sep 16 18:33:08.809202 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:08.809048    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn" event={"ID":"221247e9-8225-412e-856f-2efb8965e299","Type":"ContainerStarted","Data":"195311ceff3faedb25fac63f6c75bd420303c6ccd45648ca278a212611b85dc9"}
Sep 16 18:33:08.810859 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:08.810830    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n" event={"ID":"817087e7-6e5c-44ab-bc24-28dd0a358bd4","Type":"ContainerStarted","Data":"c443951a85012d3b63b727a246392caa5055068adc45257375cb17f7de5d95a6"}
Sep 16 18:33:08.811000 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:08.810875    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n" event={"ID":"817087e7-6e5c-44ab-bc24-28dd0a358bd4","Type":"ContainerStarted","Data":"cf3fb02f6ecdc566a2573a6b3c31ce4d2388ef45111c4607c7df3be6d91db767"}
Sep 16 18:33:08.811000 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:08.810899    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"
Sep 16 18:33:08.836728 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:08.836677    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn" podStartSLOduration=1.8366606490000001 podStartE2EDuration="1.836660649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:33:08.833768773 +0000 UTC m=+1012.392936608" watchObservedRunningTime="2026-09-16 18:33:08.836660649 +0000 UTC m=+1012.395828483"
Sep 16 18:33:08.889369 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:08.889304    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n" podStartSLOduration=1.88928116 podStartE2EDuration="1.88928116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:33:08.885859133 +0000 UTC m=+1012.445026967" watchObservedRunningTime="2026-09-16 18:33:08.88928116 +0000 UTC m=+1012.448449042"
Sep 16 18:33:08.921979 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:08.921934    2554 patch_prober.go:28] interesting pod/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 16 18:33:08.922170 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:08.922005    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r" podUID="08c849fc-98e8-46f1-b0e3-ea1a07ff4421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 16 18:33:09.922047 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:09.922004    2554 patch_prober.go:28] interesting pod/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 16 18:33:09.922502 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:09.922065    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r" podUID="08c849fc-98e8-46f1-b0e3-ea1a07ff4421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 16 18:33:10.922253 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:10.922196    2554 patch_prober.go:28] interesting pod/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 16 18:33:10.922667 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:10.922281    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r" podUID="08c849fc-98e8-46f1-b0e3-ea1a07ff4421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 16 18:33:11.136767 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.136723    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"]
Sep 16 18:33:11.137000 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.136970    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7" podUID="19c74e09-3a8d-4ba7-abfd-92e607c43d92" containerName="payload-pre-processing" containerID="cri-o://01011cafeb4fcc2a8f1586c75d52599605e20a59fd4bca9697e387b0bfa492fe" gracePeriod=30
Sep 16 18:33:11.439886 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.439816    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"
Sep 16 18:33:11.566255 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.565625    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxrv9\" (UniqueName: \"kubernetes.io/projected/19c74e09-3a8d-4ba7-abfd-92e607c43d92-kube-api-access-rxrv9\") pod \"19c74e09-3a8d-4ba7-abfd-92e607c43d92\" (UID: \"19c74e09-3a8d-4ba7-abfd-92e607c43d92\") "
Sep 16 18:33:11.566255 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.565699    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c74e09-3a8d-4ba7-abfd-92e607c43d92-plugins-config-volume\") pod \"19c74e09-3a8d-4ba7-abfd-92e607c43d92\" (UID: \"19c74e09-3a8d-4ba7-abfd-92e607c43d92\") "
Sep 16 18:33:11.566514 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.566407    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c74e09-3a8d-4ba7-abfd-92e607c43d92-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19c74e09-3a8d-4ba7-abfd-92e607c43d92" (UID: "19c74e09-3a8d-4ba7-abfd-92e607c43d92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:33:11.570257 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.569746    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c74e09-3a8d-4ba7-abfd-92e607c43d92-kube-api-access-rxrv9" (OuterVolumeSpecName: "kube-api-access-rxrv9") pod "19c74e09-3a8d-4ba7-abfd-92e607c43d92" (UID: "19c74e09-3a8d-4ba7-abfd-92e607c43d92"). InnerVolumeSpecName "kube-api-access-rxrv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:33:11.666708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.666662    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxrv9\" (UniqueName: \"kubernetes.io/projected/19c74e09-3a8d-4ba7-abfd-92e607c43d92-kube-api-access-rxrv9\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:11.666708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.666706    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c74e09-3a8d-4ba7-abfd-92e607c43d92-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:11.677056 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.677012    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"]
Sep 16 18:33:11.677317 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.677290    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r" podUID="08c849fc-98e8-46f1-b0e3-ea1a07ff4421" containerName="istio-proxy" containerID="cri-o://0cebbcde18dcc9b825701372328b3f1c4ae7fc1b15b4d202f16c20464bccd102" gracePeriod=30
Sep 16 18:33:11.827641 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.827600    2554 generic.go:356] "Generic (PLEG): container finished" podID="19c74e09-3a8d-4ba7-abfd-92e607c43d92" containerID="01011cafeb4fcc2a8f1586c75d52599605e20a59fd4bca9697e387b0bfa492fe" exitCode=0
Sep 16 18:33:11.827842 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.827650    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"
Sep 16 18:33:11.827842 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.827663    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7" event={"ID":"19c74e09-3a8d-4ba7-abfd-92e607c43d92","Type":"ContainerDied","Data":"01011cafeb4fcc2a8f1586c75d52599605e20a59fd4bca9697e387b0bfa492fe"}
Sep 16 18:33:11.827842 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.827716    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7" event={"ID":"19c74e09-3a8d-4ba7-abfd-92e607c43d92","Type":"ContainerDied","Data":"243bc833a10f6bcd27a963f74e4b3f4e8348c575e12e1545e5aa7c5f64e68db0"}
Sep 16 18:33:11.827842 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.827747    2554 scope.go:117] "RemoveContainer" containerID="01011cafeb4fcc2a8f1586c75d52599605e20a59fd4bca9697e387b0bfa492fe"
Sep 16 18:33:11.838625 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.838601    2554 scope.go:117] "RemoveContainer" containerID="01011cafeb4fcc2a8f1586c75d52599605e20a59fd4bca9697e387b0bfa492fe"
Sep 16 18:33:11.838956 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:33:11.838936    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01011cafeb4fcc2a8f1586c75d52599605e20a59fd4bca9697e387b0bfa492fe\": container with ID starting with 01011cafeb4fcc2a8f1586c75d52599605e20a59fd4bca9697e387b0bfa492fe not found: ID does not exist" containerID="01011cafeb4fcc2a8f1586c75d52599605e20a59fd4bca9697e387b0bfa492fe"
Sep 16 18:33:11.838999 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.838967    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01011cafeb4fcc2a8f1586c75d52599605e20a59fd4bca9697e387b0bfa492fe"} err="failed to get container status \"01011cafeb4fcc2a8f1586c75d52599605e20a59fd4bca9697e387b0bfa492fe\": rpc error: code = NotFound desc = could not find container \"01011cafeb4fcc2a8f1586c75d52599605e20a59fd4bca9697e387b0bfa492fe\": container with ID starting with 01011cafeb4fcc2a8f1586c75d52599605e20a59fd4bca9697e387b0bfa492fe not found: ID does not exist"
Sep 16 18:33:11.854175 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.854138    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"]
Sep 16 18:33:11.862410 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:11.862380    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7e8b5de5-6c79b8d94-mtgk7"]
Sep 16 18:33:12.884067 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:12.884027    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19c74e09-3a8d-4ba7-abfd-92e607c43d92" path="/var/lib/kubelet/pods/19c74e09-3a8d-4ba7-abfd-92e607c43d92/volumes"
Sep 16 18:33:16.654960 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:16.654919    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:33:17.037120 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.037094    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:33:17.121099 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121066    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-workload-socket\") pod \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") "
Sep 16 18:33:17.121314 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121140    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-workload-certs\") pod \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") "
Sep 16 18:33:17.121314 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121174    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istiod-ca-cert\") pod \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") "
Sep 16 18:33:17.121314 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121202    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-podinfo\") pod \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") "
Sep 16 18:33:17.121314 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121272    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-token\") pod \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") "
Sep 16 18:33:17.121532 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121315    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-envoy\") pod \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") "
Sep 16 18:33:17.121532 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121379    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08c849fc-98e8-46f1-b0e3-ea1a07ff4421" (UID: "08c849fc-98e8-46f1-b0e3-ea1a07ff4421"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:33:17.121532 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121419    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-data\") pod \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") "
Sep 16 18:33:17.121532 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121483    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-credential-socket\") pod \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") "
Sep 16 18:33:17.121532 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121528    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkzw4\" (UniqueName: \"kubernetes.io/projected/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-kube-api-access-kkzw4\") pod \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\" (UID: \"08c849fc-98e8-46f1-b0e3-ea1a07ff4421\") "
Sep 16 18:33:17.121789 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121522    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08c849fc-98e8-46f1-b0e3-ea1a07ff4421" (UID: "08c849fc-98e8-46f1-b0e3-ea1a07ff4421"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:33:17.121789 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121746    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08c849fc-98e8-46f1-b0e3-ea1a07ff4421" (UID: "08c849fc-98e8-46f1-b0e3-ea1a07ff4421"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:33:17.121789 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121605    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08c849fc-98e8-46f1-b0e3-ea1a07ff4421" (UID: "08c849fc-98e8-46f1-b0e3-ea1a07ff4421"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:33:17.121952 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121909    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:17.121952 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121933    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:17.121952 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121949    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:17.122078 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121955    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-data" (OuterVolumeSpecName: "istio-data") pod "08c849fc-98e8-46f1-b0e3-ea1a07ff4421" (UID: "08c849fc-98e8-46f1-b0e3-ea1a07ff4421"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:33:17.122078 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.121967    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:17.123592 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.123568    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08c849fc-98e8-46f1-b0e3-ea1a07ff4421" (UID: "08c849fc-98e8-46f1-b0e3-ea1a07ff4421"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:33:17.123725 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.123627    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-kube-api-access-kkzw4" (OuterVolumeSpecName: "kube-api-access-kkzw4") pod "08c849fc-98e8-46f1-b0e3-ea1a07ff4421" (UID: "08c849fc-98e8-46f1-b0e3-ea1a07ff4421"). InnerVolumeSpecName "kube-api-access-kkzw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:33:17.123900 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.123870    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08c849fc-98e8-46f1-b0e3-ea1a07ff4421" (UID: "08c849fc-98e8-46f1-b0e3-ea1a07ff4421"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:33:17.123900 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.123878    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-token" (OuterVolumeSpecName: "istio-token") pod "08c849fc-98e8-46f1-b0e3-ea1a07ff4421" (UID: "08c849fc-98e8-46f1-b0e3-ea1a07ff4421"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:33:17.222754 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.222675    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:17.222754 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.222702    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:17.222754 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.222710    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:17.222754 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.222719    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:17.222754 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.222727    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkzw4\" (UniqueName: \"kubernetes.io/projected/08c849fc-98e8-46f1-b0e3-ea1a07ff4421-kube-api-access-kkzw4\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:33:17.865302 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.865261    2554 generic.go:356] "Generic (PLEG): container finished" podID="08c849fc-98e8-46f1-b0e3-ea1a07ff4421" containerID="0cebbcde18dcc9b825701372328b3f1c4ae7fc1b15b4d202f16c20464bccd102" exitCode=0
Sep 16 18:33:17.865768 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.865319    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"
Sep 16 18:33:17.865768 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.865350    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r" event={"ID":"08c849fc-98e8-46f1-b0e3-ea1a07ff4421","Type":"ContainerDied","Data":"0cebbcde18dcc9b825701372328b3f1c4ae7fc1b15b4d202f16c20464bccd102"}
Sep 16 18:33:17.865768 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.865412    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r" event={"ID":"08c849fc-98e8-46f1-b0e3-ea1a07ff4421","Type":"ContainerDied","Data":"40e250c7022bca41948a0e2956c50eb717bd3437db5bb7b35d6c6af79cc40fc6"}
Sep 16 18:33:17.865768 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.865444    2554 scope.go:117] "RemoveContainer" containerID="0cebbcde18dcc9b825701372328b3f1c4ae7fc1b15b4d202f16c20464bccd102"
Sep 16 18:33:17.875957 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.875935    2554 scope.go:117] "RemoveContainer" containerID="b2c33528dfaa147a833781e81442f7502691fe711abe4d8e079d6c89389e8b3f"
Sep 16 18:33:17.884447 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.884431    2554 scope.go:117] "RemoveContainer" containerID="0cebbcde18dcc9b825701372328b3f1c4ae7fc1b15b4d202f16c20464bccd102"
Sep 16 18:33:17.884728 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:33:17.884710    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cebbcde18dcc9b825701372328b3f1c4ae7fc1b15b4d202f16c20464bccd102\": container with ID starting with 0cebbcde18dcc9b825701372328b3f1c4ae7fc1b15b4d202f16c20464bccd102 not found: ID does not exist" containerID="0cebbcde18dcc9b825701372328b3f1c4ae7fc1b15b4d202f16c20464bccd102"
Sep 16 18:33:17.884785 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.884736    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cebbcde18dcc9b825701372328b3f1c4ae7fc1b15b4d202f16c20464bccd102"} err="failed to get container status \"0cebbcde18dcc9b825701372328b3f1c4ae7fc1b15b4d202f16c20464bccd102\": rpc error: code = NotFound desc = could not find container \"0cebbcde18dcc9b825701372328b3f1c4ae7fc1b15b4d202f16c20464bccd102\": container with ID starting with 0cebbcde18dcc9b825701372328b3f1c4ae7fc1b15b4d202f16c20464bccd102 not found: ID does not exist"
Sep 16 18:33:17.884785 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.884751    2554 scope.go:117] "RemoveContainer" containerID="b2c33528dfaa147a833781e81442f7502691fe711abe4d8e079d6c89389e8b3f"
Sep 16 18:33:17.884998 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:33:17.884975    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c33528dfaa147a833781e81442f7502691fe711abe4d8e079d6c89389e8b3f\": container with ID starting with b2c33528dfaa147a833781e81442f7502691fe711abe4d8e079d6c89389e8b3f not found: ID does not exist" containerID="b2c33528dfaa147a833781e81442f7502691fe711abe4d8e079d6c89389e8b3f"
Sep 16 18:33:17.885060 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.885010    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c33528dfaa147a833781e81442f7502691fe711abe4d8e079d6c89389e8b3f"} err="failed to get container status \"b2c33528dfaa147a833781e81442f7502691fe711abe4d8e079d6c89389e8b3f\": rpc error: code = NotFound desc = could not find container \"b2c33528dfaa147a833781e81442f7502691fe711abe4d8e079d6c89389e8b3f\": container with ID starting with b2c33528dfaa147a833781e81442f7502691fe711abe4d8e079d6c89389e8b3f not found: ID does not exist"
Sep 16 18:33:17.908655 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.908613    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"]
Sep 16 18:33:17.912176 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:17.912143    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-7e8b5de5-openshift-default-6b87fc47dd-gdb6r"]
Sep 16 18:33:18.884058 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:18.884027    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08c849fc-98e8-46f1-b0e3-ea1a07ff4421" path="/var/lib/kubelet/pods/08c849fc-98e8-46f1-b0e3-ea1a07ff4421/volumes"
Sep 16 18:33:19.169446 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:19.169361    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:33:19.817607 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:19.817579    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"
Sep 16 18:33:19.817804 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:19.817753    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"
Sep 16 18:33:45.319115 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:33:45.319073    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:34:09.611288 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:09.611180    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:34:44.162036 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:44.161997    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:34:52.089460 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.089422    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"]
Sep 16 18:34:52.089895 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.089742    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n" podUID="817087e7-6e5c-44ab-bc24-28dd0a358bd4" containerName="payload-processing" containerID="cri-o://c443951a85012d3b63b727a246392caa5055068adc45257375cb17f7de5d95a6" gracePeriod=30
Sep 16 18:34:52.112796 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.112769    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"]
Sep 16 18:34:52.113004 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.112965    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn" podUID="221247e9-8225-412e-856f-2efb8965e299" containerName="payload-pre-processing" containerID="cri-o://b38f8c098109a1b28e96d76f2b88bd7ee884306af362b5590ca2498e9b85186f" gracePeriod=30
Sep 16 18:34:52.368465 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.368436    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"
Sep 16 18:34:52.397961 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.397931    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"
Sep 16 18:34:52.398196 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.398177    2554 generic.go:356] "Generic (PLEG): container finished" podID="817087e7-6e5c-44ab-bc24-28dd0a358bd4" containerID="c443951a85012d3b63b727a246392caa5055068adc45257375cb17f7de5d95a6" exitCode=0
Sep 16 18:34:52.398268 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.398210    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"
Sep 16 18:34:52.398328 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.398277    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n" event={"ID":"817087e7-6e5c-44ab-bc24-28dd0a358bd4","Type":"ContainerDied","Data":"c443951a85012d3b63b727a246392caa5055068adc45257375cb17f7de5d95a6"}
Sep 16 18:34:52.398389 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.398340    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n" event={"ID":"817087e7-6e5c-44ab-bc24-28dd0a358bd4","Type":"ContainerDied","Data":"cf3fb02f6ecdc566a2573a6b3c31ce4d2388ef45111c4607c7df3be6d91db767"}
Sep 16 18:34:52.398389 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.398374    2554 scope.go:117] "RemoveContainer" containerID="c443951a85012d3b63b727a246392caa5055068adc45257375cb17f7de5d95a6"
Sep 16 18:34:52.399546 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.399522    2554 generic.go:356] "Generic (PLEG): container finished" podID="221247e9-8225-412e-856f-2efb8965e299" containerID="b38f8c098109a1b28e96d76f2b88bd7ee884306af362b5590ca2498e9b85186f" exitCode=0
Sep 16 18:34:52.399633 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.399556    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn" event={"ID":"221247e9-8225-412e-856f-2efb8965e299","Type":"ContainerDied","Data":"b38f8c098109a1b28e96d76f2b88bd7ee884306af362b5590ca2498e9b85186f"}
Sep 16 18:34:52.399633 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.399608    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn" event={"ID":"221247e9-8225-412e-856f-2efb8965e299","Type":"ContainerDied","Data":"195311ceff3faedb25fac63f6c75bd420303c6ccd45648ca278a212611b85dc9"}
Sep 16 18:34:52.399633 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.399562    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"
Sep 16 18:34:52.413006 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.412980    2554 scope.go:117] "RemoveContainer" containerID="c443951a85012d3b63b727a246392caa5055068adc45257375cb17f7de5d95a6"
Sep 16 18:34:52.413435 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:34:52.413407    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c443951a85012d3b63b727a246392caa5055068adc45257375cb17f7de5d95a6\": container with ID starting with c443951a85012d3b63b727a246392caa5055068adc45257375cb17f7de5d95a6 not found: ID does not exist" containerID="c443951a85012d3b63b727a246392caa5055068adc45257375cb17f7de5d95a6"
Sep 16 18:34:52.413520 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.413443    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c443951a85012d3b63b727a246392caa5055068adc45257375cb17f7de5d95a6"} err="failed to get container status \"c443951a85012d3b63b727a246392caa5055068adc45257375cb17f7de5d95a6\": rpc error: code = NotFound desc = could not find container \"c443951a85012d3b63b727a246392caa5055068adc45257375cb17f7de5d95a6\": container with ID starting with c443951a85012d3b63b727a246392caa5055068adc45257375cb17f7de5d95a6 not found: ID does not exist"
Sep 16 18:34:52.413520 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.413461    2554 scope.go:117] "RemoveContainer" containerID="b38f8c098109a1b28e96d76f2b88bd7ee884306af362b5590ca2498e9b85186f"
Sep 16 18:34:52.422165 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.422145    2554 scope.go:117] "RemoveContainer" containerID="b38f8c098109a1b28e96d76f2b88bd7ee884306af362b5590ca2498e9b85186f"
Sep 16 18:34:52.422487 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:34:52.422467    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38f8c098109a1b28e96d76f2b88bd7ee884306af362b5590ca2498e9b85186f\": container with ID starting with b38f8c098109a1b28e96d76f2b88bd7ee884306af362b5590ca2498e9b85186f not found: ID does not exist" containerID="b38f8c098109a1b28e96d76f2b88bd7ee884306af362b5590ca2498e9b85186f"
Sep 16 18:34:52.422586 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.422497    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38f8c098109a1b28e96d76f2b88bd7ee884306af362b5590ca2498e9b85186f"} err="failed to get container status \"b38f8c098109a1b28e96d76f2b88bd7ee884306af362b5590ca2498e9b85186f\": rpc error: code = NotFound desc = could not find container \"b38f8c098109a1b28e96d76f2b88bd7ee884306af362b5590ca2498e9b85186f\": container with ID starting with b38f8c098109a1b28e96d76f2b88bd7ee884306af362b5590ca2498e9b85186f not found: ID does not exist"
Sep 16 18:34:52.465890 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.465860    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/817087e7-6e5c-44ab-bc24-28dd0a358bd4-plugins-config-volume\") pod \"817087e7-6e5c-44ab-bc24-28dd0a358bd4\" (UID: \"817087e7-6e5c-44ab-bc24-28dd0a358bd4\") "
Sep 16 18:34:52.466056 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.465963    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25jcz\" (UniqueName: \"kubernetes.io/projected/817087e7-6e5c-44ab-bc24-28dd0a358bd4-kube-api-access-25jcz\") pod \"817087e7-6e5c-44ab-bc24-28dd0a358bd4\" (UID: \"817087e7-6e5c-44ab-bc24-28dd0a358bd4\") "
Sep 16 18:34:52.466056 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.465994    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221247e9-8225-412e-856f-2efb8965e299-plugins-config-volume\") pod \"221247e9-8225-412e-856f-2efb8965e299\" (UID: \"221247e9-8225-412e-856f-2efb8965e299\") "
Sep 16 18:34:52.466149 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.466062    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2qvh\" (UniqueName: \"kubernetes.io/projected/221247e9-8225-412e-856f-2efb8965e299-kube-api-access-w2qvh\") pod \"221247e9-8225-412e-856f-2efb8965e299\" (UID: \"221247e9-8225-412e-856f-2efb8965e299\") "
Sep 16 18:34:52.466343 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.466317    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817087e7-6e5c-44ab-bc24-28dd0a358bd4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "817087e7-6e5c-44ab-bc24-28dd0a358bd4" (UID: "817087e7-6e5c-44ab-bc24-28dd0a358bd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:34:52.466449 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.466375    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221247e9-8225-412e-856f-2efb8965e299-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "221247e9-8225-412e-856f-2efb8965e299" (UID: "221247e9-8225-412e-856f-2efb8965e299"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:34:52.466508 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.466450    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/817087e7-6e5c-44ab-bc24-28dd0a358bd4-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:34:52.466508 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.466465    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221247e9-8225-412e-856f-2efb8965e299-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:34:52.468739 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.468716    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817087e7-6e5c-44ab-bc24-28dd0a358bd4-kube-api-access-25jcz" (OuterVolumeSpecName: "kube-api-access-25jcz") pod "817087e7-6e5c-44ab-bc24-28dd0a358bd4" (UID: "817087e7-6e5c-44ab-bc24-28dd0a358bd4"). InnerVolumeSpecName "kube-api-access-25jcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:34:52.468829 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.468776    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221247e9-8225-412e-856f-2efb8965e299-kube-api-access-w2qvh" (OuterVolumeSpecName: "kube-api-access-w2qvh") pod "221247e9-8225-412e-856f-2efb8965e299" (UID: "221247e9-8225-412e-856f-2efb8965e299"). InnerVolumeSpecName "kube-api-access-w2qvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:34:52.567732 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.567693    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25jcz\" (UniqueName: \"kubernetes.io/projected/817087e7-6e5c-44ab-bc24-28dd0a358bd4-kube-api-access-25jcz\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:34:52.567732 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.567725    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2qvh\" (UniqueName: \"kubernetes.io/projected/221247e9-8225-412e-856f-2efb8965e299-kube-api-access-w2qvh\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:34:52.737861 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.737822    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"]
Sep 16 18:34:52.752727 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.752692    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2c3173-5c75945489-dws9n"]
Sep 16 18:34:52.767311 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.767276    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"]
Sep 16 18:34:52.775920 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.775887    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2c3173-d95ff7594-2d6kn"]
Sep 16 18:34:52.883880 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.883841    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="221247e9-8225-412e-856f-2efb8965e299" path="/var/lib/kubelet/pods/221247e9-8225-412e-856f-2efb8965e299/volumes"
Sep 16 18:34:52.884219 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:52.884206    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="817087e7-6e5c-44ab-bc24-28dd0a358bd4" path="/var/lib/kubelet/pods/817087e7-6e5c-44ab-bc24-28dd0a358bd4/volumes"
Sep 16 18:34:59.105739 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.105693    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"]
Sep 16 18:34:59.106327 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.106309    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c849fc-98e8-46f1-b0e3-ea1a07ff4421" containerName="istio-proxy"
Sep 16 18:34:59.106327 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.106328    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c849fc-98e8-46f1-b0e3-ea1a07ff4421" containerName="istio-proxy"
Sep 16 18:34:59.106469 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.106347    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c849fc-98e8-46f1-b0e3-ea1a07ff4421" containerName="istio-proxy"
Sep 16 18:34:59.106469 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.106355    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c849fc-98e8-46f1-b0e3-ea1a07ff4421" containerName="istio-proxy"
Sep 16 18:34:59.106469 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.106365    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221247e9-8225-412e-856f-2efb8965e299" containerName="payload-pre-processing"
Sep 16 18:34:59.106469 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.106374    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="221247e9-8225-412e-856f-2efb8965e299" containerName="payload-pre-processing"
Sep 16 18:34:59.106469 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.106394    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c74e09-3a8d-4ba7-abfd-92e607c43d92" containerName="payload-pre-processing"
Sep 16 18:34:59.106469 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.106403    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c74e09-3a8d-4ba7-abfd-92e607c43d92" containerName="payload-pre-processing"
Sep 16 18:34:59.106469 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.106421    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817087e7-6e5c-44ab-bc24-28dd0a358bd4" containerName="payload-processing"
Sep 16 18:34:59.106469 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.106430    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="817087e7-6e5c-44ab-bc24-28dd0a358bd4" containerName="payload-processing"
Sep 16 18:34:59.106853 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.106521    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="08c849fc-98e8-46f1-b0e3-ea1a07ff4421" containerName="istio-proxy"
Sep 16 18:34:59.106853 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.106536    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="08c849fc-98e8-46f1-b0e3-ea1a07ff4421" containerName="istio-proxy"
Sep 16 18:34:59.106853 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.106547    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="19c74e09-3a8d-4ba7-abfd-92e607c43d92" containerName="payload-pre-processing"
Sep 16 18:34:59.106853 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.106558    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="817087e7-6e5c-44ab-bc24-28dd0a358bd4" containerName="payload-processing"
Sep 16 18:34:59.106853 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.106569    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="221247e9-8225-412e-856f-2efb8965e299" containerName="payload-pre-processing"
Sep 16 18:34:59.111525 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.111504    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.118729 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.118707    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d485ed3e-openshift-default-dockercfg-z7v6c\""
Sep 16 18:34:59.141134 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.141100    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"]
Sep 16 18:34:59.232652 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.232615    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-podinfo\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.232652 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.232655    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-workload-socket\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.232849 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.232730    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-credential-socket\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.232849 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.232749    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-token\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.232917 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.232849    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istiod-ca-cert\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.232917 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.232881    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-workload-certs\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.232982 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.232924    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbd8m\" (UniqueName: \"kubernetes.io/projected/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-kube-api-access-zbd8m\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.232982 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.232961    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-data\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.233046 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.232990    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-envoy\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.334058 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.334018    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istiod-ca-cert\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.334058 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.334057    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-workload-certs\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.334416 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.334095    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbd8m\" (UniqueName: \"kubernetes.io/projected/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-kube-api-access-zbd8m\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.334416 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.334124    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-data\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.334416 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.334148    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-envoy\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.334416 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.334174    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-podinfo\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.334416 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.334197    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-workload-socket\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.334416 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.334278    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-credential-socket\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.334416 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.334302    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-token\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.334861 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.334568    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-data\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.334861 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.334596    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-workload-socket\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.334861 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.334731    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-workload-certs\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.334861 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.334817    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-credential-socket\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.335065 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.335011    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istiod-ca-cert\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.336426 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.336403    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-envoy\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.336665 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.336645    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-podinfo\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.352986 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.352956    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbd8m\" (UniqueName: \"kubernetes.io/projected/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-kube-api-access-zbd8m\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.356935 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.356863    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-token\") pod \"e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") " pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.420748 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.420715    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:34:59.811894 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.811869    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"]
Sep 16 18:34:59.813277 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:34:59.813249    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0e5d7d_7d1a_419f_a6ed_b47a84488738.slice/crio-58ddd037fcd67344eca4720415f07a94285217088b834e9ca606d4a86aa23935 WatchSource:0}: Error finding container 58ddd037fcd67344eca4720415f07a94285217088b834e9ca606d4a86aa23935: Status 404 returned error can't find the container with id 58ddd037fcd67344eca4720415f07a94285217088b834e9ca606d4a86aa23935
Sep 16 18:34:59.815552 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.815521    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:34:59.815626 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.815593    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:34:59.815626 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:34:59.815621    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:35:00.448309 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:00.448271    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6" event={"ID":"bd0e5d7d-7d1a-419f-a6ed-b47a84488738","Type":"ContainerStarted","Data":"728fb43e4f016840b577aa00713bcbecedb3173a33936866ae7d3b4581a16926"}
Sep 16 18:35:00.448731 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:00.448325    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6" event={"ID":"bd0e5d7d-7d1a-419f-a6ed-b47a84488738","Type":"ContainerStarted","Data":"58ddd037fcd67344eca4720415f07a94285217088b834e9ca606d4a86aa23935"}
Sep 16 18:35:00.499429 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:00.499374    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6" podStartSLOduration=1.49935801 podStartE2EDuration="1.49935801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:35:00.497087356 +0000 UTC m=+1124.056255208" watchObservedRunningTime="2026-09-16 18:35:00.49935801 +0000 UTC m=+1124.058525844"
Sep 16 18:35:01.421169 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:01.421123    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:35:01.426403 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:01.426372    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:35:01.456073 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:01.456041    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:35:01.458185 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:01.458154    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:35:06.700986 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.700946    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"]
Sep 16 18:35:06.704696 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.704677    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.709285 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.709264    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-a8ab8d-openshift-default-dockercfg-ltqjc\""
Sep 16 18:35:06.804402 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.804369    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"]
Sep 16 18:35:06.810554 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.809943    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hns\" (UniqueName: \"kubernetes.io/projected/eefef367-52dd-4f0a-9f10-f2469538bd31-kube-api-access-69hns\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.810554 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.810004    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-envoy\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.810554 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.810055    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-workload-socket\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.810554 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.810086    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefef367-52dd-4f0a-9f10-f2469538bd31-istiod-ca-cert\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.810554 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.810278    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-token\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.810554 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.810330    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-workload-certs\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.810554 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.810413    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-data\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.810554 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.810449    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-credential-socket\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.810554 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.810490    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-podinfo\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.911057 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.911018    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefef367-52dd-4f0a-9f10-f2469538bd31-istiod-ca-cert\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.911463 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.911097    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-token\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.911463 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.911118    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-workload-certs\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.911463 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.911154    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-data\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.911463 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.911175    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-credential-socket\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.911463 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.911199    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-podinfo\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.911463 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.911269    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hns\" (UniqueName: \"kubernetes.io/projected/eefef367-52dd-4f0a-9f10-f2469538bd31-kube-api-access-69hns\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.911463 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.911296    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-envoy\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.911463 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.911344    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-workload-socket\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.911945 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.911916    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-credential-socket\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.911998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.911946    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefef367-52dd-4f0a-9f10-f2469538bd31-istiod-ca-cert\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.911998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.911935    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-workload-socket\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.912275 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.912255    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-data\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.912372 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.912353    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-workload-certs\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.914032 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.914000    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-envoy\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.914182 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.914163    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-podinfo\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.925757 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.925724    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-token\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:06.926016 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:06.925996    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hns\" (UniqueName: \"kubernetes.io/projected/eefef367-52dd-4f0a-9f10-f2469538bd31-kube-api-access-69hns\") pod \"e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") " pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:07.015571 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:07.015529    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:07.302885 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:07.302858    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"]
Sep 16 18:35:07.308297 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:07.308258    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:35:07.308409 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:07.308347    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:35:07.308409 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:07.308389    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:35:07.492969 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:07.492917    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd" event={"ID":"eefef367-52dd-4f0a-9f10-f2469538bd31","Type":"ContainerStarted","Data":"c8a5e36bcac4eae4d94239df145c3b5d91a7b7e6cdb3382400ac6cab365cf6bd"}
Sep 16 18:35:07.493173 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:07.492988    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd" event={"ID":"eefef367-52dd-4f0a-9f10-f2469538bd31","Type":"ContainerStarted","Data":"06e9fd56ce959b0696ed5549bed177368551fb5f12f9608c413fef222c95f315"}
Sep 16 18:35:07.521210 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:07.521143    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd" podStartSLOduration=1.521123754 podStartE2EDuration="1.521123754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:35:07.519966372 +0000 UTC m=+1131.079134208" watchObservedRunningTime="2026-09-16 18:35:07.521123754 +0000 UTC m=+1131.080291651"
Sep 16 18:35:08.015994 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:08.015949    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:08.021771 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:08.021743    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:08.498853 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:08.498825    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:08.499717 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:08.499697    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:35:14.386189 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:14.386149    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"]
Sep 16 18:35:14.390635 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:14.390609    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"
Sep 16 18:35:14.393688 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:14.393663    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d485ed3e-dockercfg-7xs9q\""
Sep 16 18:35:14.394118 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:14.394079    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d485ed3e\""
Sep 16 18:35:14.409280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:14.409256    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"]
Sep 16 18:35:14.591044 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:14.591003    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7g7j\" (UniqueName: \"kubernetes.io/projected/2c3a6de3-4539-470b-96dd-9beb6243f2c1-kube-api-access-z7g7j\") pod \"payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6\" (UID: \"2c3a6de3-4539-470b-96dd-9beb6243f2c1\") " pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"
Sep 16 18:35:14.591263 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:14.591085    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3a6de3-4539-470b-96dd-9beb6243f2c1-plugins-config-volume\") pod \"payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6\" (UID: \"2c3a6de3-4539-470b-96dd-9beb6243f2c1\") " pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"
Sep 16 18:35:14.692336 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:14.692210    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7g7j\" (UniqueName: \"kubernetes.io/projected/2c3a6de3-4539-470b-96dd-9beb6243f2c1-kube-api-access-z7g7j\") pod \"payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6\" (UID: \"2c3a6de3-4539-470b-96dd-9beb6243f2c1\") " pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"
Sep 16 18:35:14.692336 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:14.692336    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3a6de3-4539-470b-96dd-9beb6243f2c1-plugins-config-volume\") pod \"payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6\" (UID: \"2c3a6de3-4539-470b-96dd-9beb6243f2c1\") " pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"
Sep 16 18:35:14.693104 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:14.693084    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3a6de3-4539-470b-96dd-9beb6243f2c1-plugins-config-volume\") pod \"payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6\" (UID: \"2c3a6de3-4539-470b-96dd-9beb6243f2c1\") " pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"
Sep 16 18:35:14.707791 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:14.707756    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7g7j\" (UniqueName: \"kubernetes.io/projected/2c3a6de3-4539-470b-96dd-9beb6243f2c1-kube-api-access-z7g7j\") pod \"payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6\" (UID: \"2c3a6de3-4539-470b-96dd-9beb6243f2c1\") " pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"
Sep 16 18:35:15.002301 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:15.002258    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"
Sep 16 18:35:15.282818 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:15.282787    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"]
Sep 16 18:35:15.284565 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:35:15.284533    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3a6de3_4539_470b_96dd_9beb6243f2c1.slice/crio-4f6c90ff448948b2e004f04e3afd38336d8e9624b8ece241a68a4d9a264f17e6 WatchSource:0}: Error finding container 4f6c90ff448948b2e004f04e3afd38336d8e9624b8ece241a68a4d9a264f17e6: Status 404 returned error can't find the container with id 4f6c90ff448948b2e004f04e3afd38336d8e9624b8ece241a68a4d9a264f17e6
Sep 16 18:35:15.546206 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:15.546109    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6" event={"ID":"2c3a6de3-4539-470b-96dd-9beb6243f2c1","Type":"ContainerStarted","Data":"9a0c4018644f9449830f47b3c28c53ac467aecbd6bef3361a162bff0f07b5a59"}
Sep 16 18:35:15.546591 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:15.546218    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6" event={"ID":"2c3a6de3-4539-470b-96dd-9beb6243f2c1","Type":"ContainerStarted","Data":"4f6c90ff448948b2e004f04e3afd38336d8e9624b8ece241a68a4d9a264f17e6"}
Sep 16 18:35:15.546591 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:15.546260    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"
Sep 16 18:35:15.568835 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:15.568775    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6" podStartSLOduration=1.568757231 podStartE2EDuration="1.568757231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:35:15.567524727 +0000 UTC m=+1139.126692562" watchObservedRunningTime="2026-09-16 18:35:15.568757231 +0000 UTC m=+1139.127925070"
Sep 16 18:35:17.327186 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:17.327144    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:35:26.554752 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:26.554720    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"
Sep 16 18:35:43.711481 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:43.711444    2554 generic.go:356] "Generic (PLEG): container finished" podID="bd0e5d7d-7d1a-419f-a6ed-b47a84488738" containerID="728fb43e4f016840b577aa00713bcbecedb3173a33936866ae7d3b4581a16926" exitCode=0
Sep 16 18:35:43.711913 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:43.711508    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6" event={"ID":"bd0e5d7d-7d1a-419f-a6ed-b47a84488738","Type":"ContainerDied","Data":"728fb43e4f016840b577aa00713bcbecedb3173a33936866ae7d3b4581a16926"}
Sep 16 18:35:43.711913 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:43.711900    2554 scope.go:117] "RemoveContainer" containerID="728fb43e4f016840b577aa00713bcbecedb3173a33936866ae7d3b4581a16926"
Sep 16 18:35:43.713024 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:43.712994    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:35:43.713075 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:43.713064    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:35:43.713129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:43.713116    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:35:44.718743 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:44.718707    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6" event={"ID":"bd0e5d7d-7d1a-419f-a6ed-b47a84488738","Type":"ContainerStarted","Data":"6488ac901269e999e896535684078cdf9f284944f7ac4e2ef8bee51528fc9cc2"}
Sep 16 18:35:45.421778 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:45.421744    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:35:45.422652 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:45.422621    2554 patch_prober.go:28] interesting pod/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 16 18:35:45.422773 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:45.422668    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6" podUID="bd0e5d7d-7d1a-419f-a6ed-b47a84488738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 16 18:35:46.421688 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:46.421651    2554 patch_prober.go:28] interesting pod/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 16 18:35:46.422139 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:46.421709    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6" podUID="bd0e5d7d-7d1a-419f-a6ed-b47a84488738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 16 18:35:46.456370 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:46.456328    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:35:47.422044 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:47.422003    2554 patch_prober.go:28] interesting pod/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 16 18:35:47.422450 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:47.422062    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6" podUID="bd0e5d7d-7d1a-419f-a6ed-b47a84488738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 16 18:35:48.421782 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:48.421737    2554 patch_prober.go:28] interesting pod/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 16 18:35:48.422000 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:48.421820    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6" podUID="bd0e5d7d-7d1a-419f-a6ed-b47a84488738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 16 18:35:49.421334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:49.421292    2554 patch_prober.go:28] interesting pod/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 16 18:35:49.421819 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:49.421354    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6" podUID="bd0e5d7d-7d1a-419f-a6ed-b47a84488738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 16 18:35:50.425861 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:50.425826    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:35:50.426726 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:50.426707    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:35:51.524310 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:51.524275    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:35:57.218025 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.217992    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"]
Sep 16 18:35:57.218489 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.218252    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6" podUID="2c3a6de3-4539-470b-96dd-9beb6243f2c1" containerName="payload-processing" containerID="cri-o://9a0c4018644f9449830f47b3c28c53ac467aecbd6bef3361a162bff0f07b5a59" gracePeriod=30
Sep 16 18:35:57.460272 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.460245    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"
Sep 16 18:35:57.521332 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.521301    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3a6de3-4539-470b-96dd-9beb6243f2c1-plugins-config-volume\") pod \"2c3a6de3-4539-470b-96dd-9beb6243f2c1\" (UID: \"2c3a6de3-4539-470b-96dd-9beb6243f2c1\") "
Sep 16 18:35:57.521539 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.521383    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7g7j\" (UniqueName: \"kubernetes.io/projected/2c3a6de3-4539-470b-96dd-9beb6243f2c1-kube-api-access-z7g7j\") pod \"2c3a6de3-4539-470b-96dd-9beb6243f2c1\" (UID: \"2c3a6de3-4539-470b-96dd-9beb6243f2c1\") "
Sep 16 18:35:57.521675 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.521651    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3a6de3-4539-470b-96dd-9beb6243f2c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c3a6de3-4539-470b-96dd-9beb6243f2c1" (UID: "2c3a6de3-4539-470b-96dd-9beb6243f2c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:35:57.523453 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.523433    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3a6de3-4539-470b-96dd-9beb6243f2c1-kube-api-access-z7g7j" (OuterVolumeSpecName: "kube-api-access-z7g7j") pod "2c3a6de3-4539-470b-96dd-9beb6243f2c1" (UID: "2c3a6de3-4539-470b-96dd-9beb6243f2c1"). InnerVolumeSpecName "kube-api-access-z7g7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:35:57.622412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.622375    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7g7j\" (UniqueName: \"kubernetes.io/projected/2c3a6de3-4539-470b-96dd-9beb6243f2c1-kube-api-access-z7g7j\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:35:57.622412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.622406    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3a6de3-4539-470b-96dd-9beb6243f2c1-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:35:57.796180 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.796091    2554 generic.go:356] "Generic (PLEG): container finished" podID="2c3a6de3-4539-470b-96dd-9beb6243f2c1" containerID="9a0c4018644f9449830f47b3c28c53ac467aecbd6bef3361a162bff0f07b5a59" exitCode=0
Sep 16 18:35:57.796180 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.796142    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"
Sep 16 18:35:57.796180 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.796154    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6" event={"ID":"2c3a6de3-4539-470b-96dd-9beb6243f2c1","Type":"ContainerDied","Data":"9a0c4018644f9449830f47b3c28c53ac467aecbd6bef3361a162bff0f07b5a59"}
Sep 16 18:35:57.796479 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.796264    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6" event={"ID":"2c3a6de3-4539-470b-96dd-9beb6243f2c1","Type":"ContainerDied","Data":"4f6c90ff448948b2e004f04e3afd38336d8e9624b8ece241a68a4d9a264f17e6"}
Sep 16 18:35:57.796479 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.796297    2554 scope.go:117] "RemoveContainer" containerID="9a0c4018644f9449830f47b3c28c53ac467aecbd6bef3361a162bff0f07b5a59"
Sep 16 18:35:57.806506 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.806486    2554 scope.go:117] "RemoveContainer" containerID="9a0c4018644f9449830f47b3c28c53ac467aecbd6bef3361a162bff0f07b5a59"
Sep 16 18:35:57.806848 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:35:57.806815    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0c4018644f9449830f47b3c28c53ac467aecbd6bef3361a162bff0f07b5a59\": container with ID starting with 9a0c4018644f9449830f47b3c28c53ac467aecbd6bef3361a162bff0f07b5a59 not found: ID does not exist" containerID="9a0c4018644f9449830f47b3c28c53ac467aecbd6bef3361a162bff0f07b5a59"
Sep 16 18:35:57.806893 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.806861    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0c4018644f9449830f47b3c28c53ac467aecbd6bef3361a162bff0f07b5a59"} err="failed to get container status \"9a0c4018644f9449830f47b3c28c53ac467aecbd6bef3361a162bff0f07b5a59\": rpc error: code = NotFound desc = could not find container \"9a0c4018644f9449830f47b3c28c53ac467aecbd6bef3361a162bff0f07b5a59\": container with ID starting with 9a0c4018644f9449830f47b3c28c53ac467aecbd6bef3361a162bff0f07b5a59 not found: ID does not exist"
Sep 16 18:35:57.824111 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.824082    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"]
Sep 16 18:35:57.827223 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:57.827199    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d485ed3e-58c95c4985-sb4x6"]
Sep 16 18:35:58.884343 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:35:58.884307    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c3a6de3-4539-470b-96dd-9beb6243f2c1" path="/var/lib/kubelet/pods/2c3a6de3-4539-470b-96dd-9beb6243f2c1/volumes"
Sep 16 18:36:04.593656 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:04.593613    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"]
Sep 16 18:36:04.594122 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:04.593874    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6" podUID="bd0e5d7d-7d1a-419f-a6ed-b47a84488738" containerName="istio-proxy" containerID="cri-o://6488ac901269e999e896535684078cdf9f284944f7ac4e2ef8bee51528fc9cc2" gracePeriod=30
Sep 16 18:36:05.428851 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:05.428816    2554 patch_prober.go:28] interesting pod/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 16 18:36:05.429022 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:05.428890    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6" podUID="bd0e5d7d-7d1a-419f-a6ed-b47a84488738" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:36:09.844881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.844851    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:36:09.865609 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.865536    2554 generic.go:356] "Generic (PLEG): container finished" podID="bd0e5d7d-7d1a-419f-a6ed-b47a84488738" containerID="6488ac901269e999e896535684078cdf9f284944f7ac4e2ef8bee51528fc9cc2" exitCode=0
Sep 16 18:36:09.865609 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.865584    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"
Sep 16 18:36:09.865776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.865615    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6" event={"ID":"bd0e5d7d-7d1a-419f-a6ed-b47a84488738","Type":"ContainerDied","Data":"6488ac901269e999e896535684078cdf9f284944f7ac4e2ef8bee51528fc9cc2"}
Sep 16 18:36:09.865776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.865666    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6" event={"ID":"bd0e5d7d-7d1a-419f-a6ed-b47a84488738","Type":"ContainerDied","Data":"58ddd037fcd67344eca4720415f07a94285217088b834e9ca606d4a86aa23935"}
Sep 16 18:36:09.865776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.865688    2554 scope.go:117] "RemoveContainer" containerID="6488ac901269e999e896535684078cdf9f284944f7ac4e2ef8bee51528fc9cc2"
Sep 16 18:36:09.877089 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.877059    2554 scope.go:117] "RemoveContainer" containerID="728fb43e4f016840b577aa00713bcbecedb3173a33936866ae7d3b4581a16926"
Sep 16 18:36:09.889216 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.889188    2554 scope.go:117] "RemoveContainer" containerID="6488ac901269e999e896535684078cdf9f284944f7ac4e2ef8bee51528fc9cc2"
Sep 16 18:36:09.889589 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:36:09.889569    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6488ac901269e999e896535684078cdf9f284944f7ac4e2ef8bee51528fc9cc2\": container with ID starting with 6488ac901269e999e896535684078cdf9f284944f7ac4e2ef8bee51528fc9cc2 not found: ID does not exist" containerID="6488ac901269e999e896535684078cdf9f284944f7ac4e2ef8bee51528fc9cc2"
Sep 16 18:36:09.889646 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.889600    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6488ac901269e999e896535684078cdf9f284944f7ac4e2ef8bee51528fc9cc2"} err="failed to get container status \"6488ac901269e999e896535684078cdf9f284944f7ac4e2ef8bee51528fc9cc2\": rpc error: code = NotFound desc = could not find container \"6488ac901269e999e896535684078cdf9f284944f7ac4e2ef8bee51528fc9cc2\": container with ID starting with 6488ac901269e999e896535684078cdf9f284944f7ac4e2ef8bee51528fc9cc2 not found: ID does not exist"
Sep 16 18:36:09.889646 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.889617    2554 scope.go:117] "RemoveContainer" containerID="728fb43e4f016840b577aa00713bcbecedb3173a33936866ae7d3b4581a16926"
Sep 16 18:36:09.889917 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:36:09.889897    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728fb43e4f016840b577aa00713bcbecedb3173a33936866ae7d3b4581a16926\": container with ID starting with 728fb43e4f016840b577aa00713bcbecedb3173a33936866ae7d3b4581a16926 not found: ID does not exist" containerID="728fb43e4f016840b577aa00713bcbecedb3173a33936866ae7d3b4581a16926"
Sep 16 18:36:09.889976 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.889929    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728fb43e4f016840b577aa00713bcbecedb3173a33936866ae7d3b4581a16926"} err="failed to get container status \"728fb43e4f016840b577aa00713bcbecedb3173a33936866ae7d3b4581a16926\": rpc error: code = NotFound desc = could not find container \"728fb43e4f016840b577aa00713bcbecedb3173a33936866ae7d3b4581a16926\": container with ID starting with 728fb43e4f016840b577aa00713bcbecedb3173a33936866ae7d3b4581a16926 not found: ID does not exist"
Sep 16 18:36:09.953281 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.953222    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-workload-socket\") pod \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") "
Sep 16 18:36:09.953281 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.953283    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-credential-socket\") pod \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") "
Sep 16 18:36:09.953500 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.953310    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbd8m\" (UniqueName: \"kubernetes.io/projected/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-kube-api-access-zbd8m\") pod \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") "
Sep 16 18:36:09.953500 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.953355    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-data\") pod \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") "
Sep 16 18:36:09.953500 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.953419    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-envoy\") pod \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") "
Sep 16 18:36:09.953500 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.953453    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istiod-ca-cert\") pod \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") "
Sep 16 18:36:09.953500 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.953479    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bd0e5d7d-7d1a-419f-a6ed-b47a84488738" (UID: "bd0e5d7d-7d1a-419f-a6ed-b47a84488738"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:09.953703 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.953637    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bd0e5d7d-7d1a-419f-a6ed-b47a84488738" (UID: "bd0e5d7d-7d1a-419f-a6ed-b47a84488738"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:09.953703 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.953491    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-token\") pod \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") "
Sep 16 18:36:09.953703 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.953662    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-data" (OuterVolumeSpecName: "istio-data") pod "bd0e5d7d-7d1a-419f-a6ed-b47a84488738" (UID: "bd0e5d7d-7d1a-419f-a6ed-b47a84488738"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:09.953852 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.953767    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-podinfo\") pod \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") "
Sep 16 18:36:09.953852 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.953834    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-workload-certs\") pod \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\" (UID: \"bd0e5d7d-7d1a-419f-a6ed-b47a84488738\") "
Sep 16 18:36:09.953946 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.953845    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bd0e5d7d-7d1a-419f-a6ed-b47a84488738" (UID: "bd0e5d7d-7d1a-419f-a6ed-b47a84488738"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:36:09.954127 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.954080    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bd0e5d7d-7d1a-419f-a6ed-b47a84488738" (UID: "bd0e5d7d-7d1a-419f-a6ed-b47a84488738"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:09.954602 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.954354    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:09.954602 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.954379    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:09.954602 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.954396    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:09.954602 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.954409    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:09.954602 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.954422    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:09.955723 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.955697    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-kube-api-access-zbd8m" (OuterVolumeSpecName: "kube-api-access-zbd8m") pod "bd0e5d7d-7d1a-419f-a6ed-b47a84488738" (UID: "bd0e5d7d-7d1a-419f-a6ed-b47a84488738"). InnerVolumeSpecName "kube-api-access-zbd8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:36:09.955821 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.955785    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-token" (OuterVolumeSpecName: "istio-token") pod "bd0e5d7d-7d1a-419f-a6ed-b47a84488738" (UID: "bd0e5d7d-7d1a-419f-a6ed-b47a84488738"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:36:09.956082 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.956060    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bd0e5d7d-7d1a-419f-a6ed-b47a84488738" (UID: "bd0e5d7d-7d1a-419f-a6ed-b47a84488738"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:36:09.956134 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:09.956062    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bd0e5d7d-7d1a-419f-a6ed-b47a84488738" (UID: "bd0e5d7d-7d1a-419f-a6ed-b47a84488738"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:10.055733 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:10.055689    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:10.055733 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:10.055728    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbd8m\" (UniqueName: \"kubernetes.io/projected/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-kube-api-access-zbd8m\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:10.055733 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:10.055741    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:10.055972 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:10.055753    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0e5d7d-7d1a-419f-a6ed-b47a84488738-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:10.192277 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:10.192244    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"]
Sep 16 18:36:10.197263 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:10.197220    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d485ed3e-openshift-default-7f845696f6-c5nn6"]
Sep 16 18:36:10.883874 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:10.883831    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd0e5d7d-7d1a-419f-a6ed-b47a84488738" path="/var/lib/kubelet/pods/bd0e5d7d-7d1a-419f-a6ed-b47a84488738/volumes"
Sep 16 18:36:13.941269 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:13.941220    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"]
Sep 16 18:36:13.941716 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:13.941498    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd" podUID="eefef367-52dd-4f0a-9f10-f2469538bd31" containerName="istio-proxy" containerID="cri-o://c8a5e36bcac4eae4d94239df145c3b5d91a7b7e6cdb3382400ac6cab365cf6bd" gracePeriod=30
Sep 16 18:36:16.861375 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:16.861344    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:36:16.865428 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:16.865407    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:36:19.172667 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.172643    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:36:19.191265 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.191241    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:36:19.352907 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.352870    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-token\") pod \"eefef367-52dd-4f0a-9f10-f2469538bd31\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") "
Sep 16 18:36:19.353110 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.352939    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-envoy\") pod \"eefef367-52dd-4f0a-9f10-f2469538bd31\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") "
Sep 16 18:36:19.353110 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.352996    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefef367-52dd-4f0a-9f10-f2469538bd31-istiod-ca-cert\") pod \"eefef367-52dd-4f0a-9f10-f2469538bd31\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") "
Sep 16 18:36:19.353110 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.353042    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69hns\" (UniqueName: \"kubernetes.io/projected/eefef367-52dd-4f0a-9f10-f2469538bd31-kube-api-access-69hns\") pod \"eefef367-52dd-4f0a-9f10-f2469538bd31\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") "
Sep 16 18:36:19.353110 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.353071    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-workload-certs\") pod \"eefef367-52dd-4f0a-9f10-f2469538bd31\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") "
Sep 16 18:36:19.353110 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.353095    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-podinfo\") pod \"eefef367-52dd-4f0a-9f10-f2469538bd31\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") "
Sep 16 18:36:19.353394 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.353151    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-workload-socket\") pod \"eefef367-52dd-4f0a-9f10-f2469538bd31\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") "
Sep 16 18:36:19.353394 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.353210    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-data\") pod \"eefef367-52dd-4f0a-9f10-f2469538bd31\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") "
Sep 16 18:36:19.353394 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.353279    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-credential-socket\") pod \"eefef367-52dd-4f0a-9f10-f2469538bd31\" (UID: \"eefef367-52dd-4f0a-9f10-f2469538bd31\") "
Sep 16 18:36:19.353394 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.353364    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eefef367-52dd-4f0a-9f10-f2469538bd31" (UID: "eefef367-52dd-4f0a-9f10-f2469538bd31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:19.353580 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.353426    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eefef367-52dd-4f0a-9f10-f2469538bd31-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eefef367-52dd-4f0a-9f10-f2469538bd31" (UID: "eefef367-52dd-4f0a-9f10-f2469538bd31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:36:19.353580 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.353564    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-data" (OuterVolumeSpecName: "istio-data") pod "eefef367-52dd-4f0a-9f10-f2469538bd31" (UID: "eefef367-52dd-4f0a-9f10-f2469538bd31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:19.353683 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.353588    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eefef367-52dd-4f0a-9f10-f2469538bd31" (UID: "eefef367-52dd-4f0a-9f10-f2469538bd31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:19.353683 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.353617    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eefef367-52dd-4f0a-9f10-f2469538bd31" (UID: "eefef367-52dd-4f0a-9f10-f2469538bd31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:19.353781 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.353705    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:19.353781 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.353724    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:19.353781 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.353739    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefef367-52dd-4f0a-9f10-f2469538bd31-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:19.353781 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.353752    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:19.353781 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.353765    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:19.355184 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.355160    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eefef367-52dd-4f0a-9f10-f2469538bd31" (UID: "eefef367-52dd-4f0a-9f10-f2469538bd31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:19.355548 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.355528    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-token" (OuterVolumeSpecName: "istio-token") pod "eefef367-52dd-4f0a-9f10-f2469538bd31" (UID: "eefef367-52dd-4f0a-9f10-f2469538bd31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:36:19.355626 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.355556    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefef367-52dd-4f0a-9f10-f2469538bd31-kube-api-access-69hns" (OuterVolumeSpecName: "kube-api-access-69hns") pod "eefef367-52dd-4f0a-9f10-f2469538bd31" (UID: "eefef367-52dd-4f0a-9f10-f2469538bd31"). InnerVolumeSpecName "kube-api-access-69hns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:36:19.355757 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.355739    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eefef367-52dd-4f0a-9f10-f2469538bd31" (UID: "eefef367-52dd-4f0a-9f10-f2469538bd31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:36:19.454997 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.454958    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69hns\" (UniqueName: \"kubernetes.io/projected/eefef367-52dd-4f0a-9f10-f2469538bd31-kube-api-access-69hns\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:19.454997 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.454990    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:19.454997 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.455000    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:19.454997 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.455009    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefef367-52dd-4f0a-9f10-f2469538bd31-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:36:19.919983 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.919948    2554 generic.go:356] "Generic (PLEG): container finished" podID="eefef367-52dd-4f0a-9f10-f2469538bd31" containerID="c8a5e36bcac4eae4d94239df145c3b5d91a7b7e6cdb3382400ac6cab365cf6bd" exitCode=0
Sep 16 18:36:19.920157 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.919989    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"
Sep 16 18:36:19.920157 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.920032    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd" event={"ID":"eefef367-52dd-4f0a-9f10-f2469538bd31","Type":"ContainerDied","Data":"c8a5e36bcac4eae4d94239df145c3b5d91a7b7e6cdb3382400ac6cab365cf6bd"}
Sep 16 18:36:19.920157 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.920092    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd" event={"ID":"eefef367-52dd-4f0a-9f10-f2469538bd31","Type":"ContainerDied","Data":"06e9fd56ce959b0696ed5549bed177368551fb5f12f9608c413fef222c95f315"}
Sep 16 18:36:19.920157 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.920119    2554 scope.go:117] "RemoveContainer" containerID="c8a5e36bcac4eae4d94239df145c3b5d91a7b7e6cdb3382400ac6cab365cf6bd"
Sep 16 18:36:19.930042 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.930024    2554 scope.go:117] "RemoveContainer" containerID="c8a5e36bcac4eae4d94239df145c3b5d91a7b7e6cdb3382400ac6cab365cf6bd"
Sep 16 18:36:19.930309 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:36:19.930291    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a5e36bcac4eae4d94239df145c3b5d91a7b7e6cdb3382400ac6cab365cf6bd\": container with ID starting with c8a5e36bcac4eae4d94239df145c3b5d91a7b7e6cdb3382400ac6cab365cf6bd not found: ID does not exist" containerID="c8a5e36bcac4eae4d94239df145c3b5d91a7b7e6cdb3382400ac6cab365cf6bd"
Sep 16 18:36:19.930363 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.930317    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a5e36bcac4eae4d94239df145c3b5d91a7b7e6cdb3382400ac6cab365cf6bd"} err="failed to get container status \"c8a5e36bcac4eae4d94239df145c3b5d91a7b7e6cdb3382400ac6cab365cf6bd\": rpc error: code = NotFound desc = could not find container \"c8a5e36bcac4eae4d94239df145c3b5d91a7b7e6cdb3382400ac6cab365cf6bd\": container with ID starting with c8a5e36bcac4eae4d94239df145c3b5d91a7b7e6cdb3382400ac6cab365cf6bd not found: ID does not exist"
Sep 16 18:36:19.950623 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.950593    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"]
Sep 16 18:36:19.960745 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:19.960716    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-a8ab8d-openshift-default-5d97d54fc6-wvrsd"]
Sep 16 18:36:20.322780 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.322749    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"]
Sep 16 18:36:20.323203 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.323187    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd0e5d7d-7d1a-419f-a6ed-b47a84488738" containerName="istio-proxy"
Sep 16 18:36:20.323260 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.323204    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd0e5d7d-7d1a-419f-a6ed-b47a84488738" containerName="istio-proxy"
Sep 16 18:36:20.323260 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.323223    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd0e5d7d-7d1a-419f-a6ed-b47a84488738" containerName="istio-proxy"
Sep 16 18:36:20.323260 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.323242    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd0e5d7d-7d1a-419f-a6ed-b47a84488738" containerName="istio-proxy"
Sep 16 18:36:20.323260 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.323255    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c3a6de3-4539-470b-96dd-9beb6243f2c1" containerName="payload-processing"
Sep 16 18:36:20.323260 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.323261    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c3a6de3-4539-470b-96dd-9beb6243f2c1" containerName="payload-processing"
Sep 16 18:36:20.323420 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.323276    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eefef367-52dd-4f0a-9f10-f2469538bd31" containerName="istio-proxy"
Sep 16 18:36:20.323420 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.323281    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="eefef367-52dd-4f0a-9f10-f2469538bd31" containerName="istio-proxy"
Sep 16 18:36:20.323420 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.323366    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c3a6de3-4539-470b-96dd-9beb6243f2c1" containerName="payload-processing"
Sep 16 18:36:20.323420 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.323377    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="eefef367-52dd-4f0a-9f10-f2469538bd31" containerName="istio-proxy"
Sep 16 18:36:20.323420 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.323383    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd0e5d7d-7d1a-419f-a6ed-b47a84488738" containerName="istio-proxy"
Sep 16 18:36:20.323420 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.323390    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd0e5d7d-7d1a-419f-a6ed-b47a84488738" containerName="istio-proxy"
Sep 16 18:36:20.326307 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.326286    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.329541 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.329522    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-ad6826-openshift-default-dockercfg-6txqs\""
Sep 16 18:36:20.353870 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.353831    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"]
Sep 16 18:36:20.464008 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.463976    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-envoy\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.464008 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.464011    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-token\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.464222 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.464045    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-podinfo\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.464222 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.464070    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-workload-socket\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.464222 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.464159    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b2ef39-794f-4860-9a43-a3ba52fcb217-istiod-ca-cert\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.464339 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.464255    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-credential-socket\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.464339 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.464289    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-data\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.464339 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.464329    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-workload-certs\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.464445 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.464428    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtl2c\" (UniqueName: \"kubernetes.io/projected/94b2ef39-794f-4860-9a43-a3ba52fcb217-kube-api-access-wtl2c\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.565127 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.565082    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-data\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.565356 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.565143    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-workload-certs\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.565356 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.565201    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtl2c\" (UniqueName: \"kubernetes.io/projected/94b2ef39-794f-4860-9a43-a3ba52fcb217-kube-api-access-wtl2c\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.565356 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.565258    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-envoy\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.565356 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.565283    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-token\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.565356 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.565321    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-podinfo\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.565607 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.565369    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-workload-socket\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.565607 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.565409    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b2ef39-794f-4860-9a43-a3ba52fcb217-istiod-ca-cert\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.565607 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.565463    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-credential-socket\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.565607 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.565552    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-data\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.565769 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.565753    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-workload-socket\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.565894 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.565873    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-workload-certs\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.566141 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.566108    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-credential-socket\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.566287 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.566182    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b2ef39-794f-4860-9a43-a3ba52fcb217-istiod-ca-cert\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.567627 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.567605    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-envoy\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.567942 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.567919    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-podinfo\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.578145 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.578084    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtl2c\" (UniqueName: \"kubernetes.io/projected/94b2ef39-794f-4860-9a43-a3ba52fcb217-kube-api-access-wtl2c\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.590752 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.590725    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-token\") pod \"e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") " pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.636512 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.636469    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:20.884125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.884034    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eefef367-52dd-4f0a-9f10-f2469538bd31" path="/var/lib/kubelet/pods/eefef367-52dd-4f0a-9f10-f2469538bd31/volumes"
Sep 16 18:36:20.979251 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.979206    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"]
Sep 16 18:36:20.981750 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:36:20.981722    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b2ef39_794f_4860_9a43_a3ba52fcb217.slice/crio-75eb9d27d8c9df09fc0ea757af72cb43b74662cc7fea0a946383a34fdf0e2c73 WatchSource:0}: Error finding container 75eb9d27d8c9df09fc0ea757af72cb43b74662cc7fea0a946383a34fdf0e2c73: Status 404 returned error can't find the container with id 75eb9d27d8c9df09fc0ea757af72cb43b74662cc7fea0a946383a34fdf0e2c73
Sep 16 18:36:20.983921 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.983887    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:36:20.983984 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.983956    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:36:20.984024 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:20.983984    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:36:21.933625 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:21.933592    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4" event={"ID":"94b2ef39-794f-4860-9a43-a3ba52fcb217","Type":"ContainerStarted","Data":"d1f389915d6f95bcd9b153d5ff93685693cc719b21b09bcea4a9421006cd8dbd"}
Sep 16 18:36:21.934072 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:21.933641    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4" event={"ID":"94b2ef39-794f-4860-9a43-a3ba52fcb217","Type":"ContainerStarted","Data":"75eb9d27d8c9df09fc0ea757af72cb43b74662cc7fea0a946383a34fdf0e2c73"}
Sep 16 18:36:21.964512 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:21.964453    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4" podStartSLOduration=1.96443893 podStartE2EDuration="1.96443893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:36:21.96137096 +0000 UTC m=+1205.520538794" watchObservedRunningTime="2026-09-16 18:36:21.96443893 +0000 UTC m=+1205.523606762"
Sep 16 18:36:22.636853 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:22.636814    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:22.641792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:22.641767    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:22.941790 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:22.941701    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:22.942916 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:22.942892    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:29.119338 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.119305    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"]
Sep 16 18:36:29.123243 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.123212    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"
Sep 16 18:36:29.126199 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.126174    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-ad6826-dockercfg-bcwhx\""
Sep 16 18:36:29.127623 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.127602    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-ad6826\""
Sep 16 18:36:29.136075 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.136050    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"]
Sep 16 18:36:29.203003 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.202973    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"]
Sep 16 18:36:29.206774 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.206755    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"
Sep 16 18:36:29.219601 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.219569    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"]
Sep 16 18:36:29.248661 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.248623    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sh2z\" (UniqueName: \"kubernetes.io/projected/3ef3ad01-7823-466a-a3a4-274395e56a75-kube-api-access-2sh2z\") pod \"payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf\" (UID: \"3ef3ad01-7823-466a-a3a4-274395e56a75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"
Sep 16 18:36:29.248835 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.248675    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef3ad01-7823-466a-a3a4-274395e56a75-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf\" (UID: \"3ef3ad01-7823-466a-a3a4-274395e56a75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"
Sep 16 18:36:29.248835 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.248737    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c34855-8f87-4fca-85b6-ac474f5a8359-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25\" (UID: \"e4c34855-8f87-4fca-85b6-ac474f5a8359\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"
Sep 16 18:36:29.248835 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.248825    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfl5x\" (UniqueName: \"kubernetes.io/projected/e4c34855-8f87-4fca-85b6-ac474f5a8359-kube-api-access-xfl5x\") pod \"payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25\" (UID: \"e4c34855-8f87-4fca-85b6-ac474f5a8359\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"
Sep 16 18:36:29.349462 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.349420    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sh2z\" (UniqueName: \"kubernetes.io/projected/3ef3ad01-7823-466a-a3a4-274395e56a75-kube-api-access-2sh2z\") pod \"payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf\" (UID: \"3ef3ad01-7823-466a-a3a4-274395e56a75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"
Sep 16 18:36:29.349660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.349473    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef3ad01-7823-466a-a3a4-274395e56a75-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf\" (UID: \"3ef3ad01-7823-466a-a3a4-274395e56a75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"
Sep 16 18:36:29.349660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.349520    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c34855-8f87-4fca-85b6-ac474f5a8359-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25\" (UID: \"e4c34855-8f87-4fca-85b6-ac474f5a8359\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"
Sep 16 18:36:29.349660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.349586    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfl5x\" (UniqueName: \"kubernetes.io/projected/e4c34855-8f87-4fca-85b6-ac474f5a8359-kube-api-access-xfl5x\") pod \"payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25\" (UID: \"e4c34855-8f87-4fca-85b6-ac474f5a8359\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"
Sep 16 18:36:29.350447 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.350423    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c34855-8f87-4fca-85b6-ac474f5a8359-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25\" (UID: \"e4c34855-8f87-4fca-85b6-ac474f5a8359\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"
Sep 16 18:36:29.350504 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.350479    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef3ad01-7823-466a-a3a4-274395e56a75-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf\" (UID: \"3ef3ad01-7823-466a-a3a4-274395e56a75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"
Sep 16 18:36:29.380775 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.380305    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sh2z\" (UniqueName: \"kubernetes.io/projected/3ef3ad01-7823-466a-a3a4-274395e56a75-kube-api-access-2sh2z\") pod \"payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf\" (UID: \"3ef3ad01-7823-466a-a3a4-274395e56a75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"
Sep 16 18:36:29.380775 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.380645    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfl5x\" (UniqueName: \"kubernetes.io/projected/e4c34855-8f87-4fca-85b6-ac474f5a8359-kube-api-access-xfl5x\") pod \"payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25\" (UID: \"e4c34855-8f87-4fca-85b6-ac474f5a8359\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"
Sep 16 18:36:29.434403 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.434362    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"
Sep 16 18:36:29.517015 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.516984    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"
Sep 16 18:36:29.730551 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.730522    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"]
Sep 16 18:36:29.732629 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:36:29.732599    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c34855_8f87_4fca_85b6_ac474f5a8359.slice/crio-190830389e494e7bbd72ea479097a42e8963ca0500436959908466ce1b826db9 WatchSource:0}: Error finding container 190830389e494e7bbd72ea479097a42e8963ca0500436959908466ce1b826db9: Status 404 returned error can't find the container with id 190830389e494e7bbd72ea479097a42e8963ca0500436959908466ce1b826db9
Sep 16 18:36:29.812291 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.812265    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"]
Sep 16 18:36:29.814821 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:36:29.814794    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef3ad01_7823_466a_a3a4_274395e56a75.slice/crio-8d564758cd04c6ec884f45eff2f86018a0491646ff01a2c15817a9e29fdd8af1 WatchSource:0}: Error finding container 8d564758cd04c6ec884f45eff2f86018a0491646ff01a2c15817a9e29fdd8af1: Status 404 returned error can't find the container with id 8d564758cd04c6ec884f45eff2f86018a0491646ff01a2c15817a9e29fdd8af1
Sep 16 18:36:29.985859 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.985762    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf" event={"ID":"3ef3ad01-7823-466a-a3a4-274395e56a75","Type":"ContainerStarted","Data":"857536a6bb015da6bda3434931423d0b232cee9d6f210506f99a65ec5863c844"}
Sep 16 18:36:29.985859 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.985819    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf" event={"ID":"3ef3ad01-7823-466a-a3a4-274395e56a75","Type":"ContainerStarted","Data":"8d564758cd04c6ec884f45eff2f86018a0491646ff01a2c15817a9e29fdd8af1"}
Sep 16 18:36:29.986085 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.985865    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"
Sep 16 18:36:29.987585 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.987558    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25" event={"ID":"e4c34855-8f87-4fca-85b6-ac474f5a8359","Type":"ContainerStarted","Data":"7321101d0b08c64c0e20e563580ef02a61373e4474984e1fc478dbaac503aaeb"}
Sep 16 18:36:29.987718 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.987606    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25" event={"ID":"e4c34855-8f87-4fca-85b6-ac474f5a8359","Type":"ContainerStarted","Data":"190830389e494e7bbd72ea479097a42e8963ca0500436959908466ce1b826db9"}
Sep 16 18:36:29.987718 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:29.987634    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"
Sep 16 18:36:30.019383 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:30.019331    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf" podStartSLOduration=1.019314439 podStartE2EDuration="1.019314439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:36:30.017443064 +0000 UTC m=+1213.576610899" watchObservedRunningTime="2026-09-16 18:36:30.019314439 +0000 UTC m=+1213.578482272"
Sep 16 18:36:30.070769 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:30.070717    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25" podStartSLOduration=1.070704226 podStartE2EDuration="1.070704226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:36:30.068172883 +0000 UTC m=+1213.627340718" watchObservedRunningTime="2026-09-16 18:36:30.070704226 +0000 UTC m=+1213.629872058"
Sep 16 18:36:40.996312 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:40.996273    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"
Sep 16 18:36:40.996816 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:40.996375    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"
Sep 16 18:36:46.081814 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:46.081775    2554 generic.go:356] "Generic (PLEG): container finished" podID="94b2ef39-794f-4860-9a43-a3ba52fcb217" containerID="d1f389915d6f95bcd9b153d5ff93685693cc719b21b09bcea4a9421006cd8dbd" exitCode=0
Sep 16 18:36:46.082259 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:46.081845    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4" event={"ID":"94b2ef39-794f-4860-9a43-a3ba52fcb217","Type":"ContainerDied","Data":"d1f389915d6f95bcd9b153d5ff93685693cc719b21b09bcea4a9421006cd8dbd"}
Sep 16 18:36:46.082316 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:46.082275    2554 scope.go:117] "RemoveContainer" containerID="d1f389915d6f95bcd9b153d5ff93685693cc719b21b09bcea4a9421006cd8dbd"
Sep 16 18:36:46.083391 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:46.083361    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:36:46.083455 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:46.083431    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:36:46.083496 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:46.083463    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:36:47.089036 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:47.088991    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4" event={"ID":"94b2ef39-794f-4860-9a43-a3ba52fcb217","Type":"ContainerStarted","Data":"92fa280136b69c61f6b6b21ae97ffce70e9daadbf6587c4876a2f318697d8fdc"}
Sep 16 18:36:47.636696 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:47.636644    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:47.636999 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:47.636975    2554 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 18:36:47.637063 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:47.637026    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4" podUID="94b2ef39-794f-4860-9a43-a3ba52fcb217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 18:36:48.636948 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:48.636914    2554 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 18:36:48.637411 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:48.636972    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4" podUID="94b2ef39-794f-4860-9a43-a3ba52fcb217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 18:36:49.637567 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:49.637525    2554 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 18:36:49.638040 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:49.637622    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4" podUID="94b2ef39-794f-4860-9a43-a3ba52fcb217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 18:36:50.636940 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:50.636892    2554 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 18:36:50.637151 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:50.636961    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4" podUID="94b2ef39-794f-4860-9a43-a3ba52fcb217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 18:36:51.637346 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:51.637306    2554 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 18:36:51.637766 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:51.637376    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4" podUID="94b2ef39-794f-4860-9a43-a3ba52fcb217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 18:36:52.641139 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:52.641095    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:52.641623 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:52.641610    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:52.642447 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:52.642426    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:36:55.363002 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.362956    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"]
Sep 16 18:36:55.366661 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.366637    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"
Sep 16 18:36:55.368876 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.368848    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-cf62d1\""
Sep 16 18:36:55.369019 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.368940    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-cf62d1-dockercfg-vfx4c\""
Sep 16 18:36:55.378885 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.378860    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"]
Sep 16 18:36:55.406222 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.406190    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"]
Sep 16 18:36:55.409761 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.409741    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"
Sep 16 18:36:55.423146 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.423121    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"]
Sep 16 18:36:55.525779 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.525744    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kszh\" (UniqueName: \"kubernetes.io/projected/22f4dc05-42f8-4fc7-8543-15cefc4e80c7-kube-api-access-2kszh\") pod \"payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw\" (UID: \"22f4dc05-42f8-4fc7-8543-15cefc4e80c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"
Sep 16 18:36:55.525975 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.525790    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655f67e1-9da5-4ccf-b57c-c085d8bb523e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv\" (UID: \"655f67e1-9da5-4ccf-b57c-c085d8bb523e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"
Sep 16 18:36:55.525975 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.525896    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd4jp\" (UniqueName: \"kubernetes.io/projected/655f67e1-9da5-4ccf-b57c-c085d8bb523e-kube-api-access-bd4jp\") pod \"payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv\" (UID: \"655f67e1-9da5-4ccf-b57c-c085d8bb523e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"
Sep 16 18:36:55.526103 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.525970    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f4dc05-42f8-4fc7-8543-15cefc4e80c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw\" (UID: \"22f4dc05-42f8-4fc7-8543-15cefc4e80c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"
Sep 16 18:36:55.627432 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.627330    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kszh\" (UniqueName: \"kubernetes.io/projected/22f4dc05-42f8-4fc7-8543-15cefc4e80c7-kube-api-access-2kszh\") pod \"payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw\" (UID: \"22f4dc05-42f8-4fc7-8543-15cefc4e80c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"
Sep 16 18:36:55.627432 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.627383    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655f67e1-9da5-4ccf-b57c-c085d8bb523e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv\" (UID: \"655f67e1-9da5-4ccf-b57c-c085d8bb523e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"
Sep 16 18:36:55.627432 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.627425    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd4jp\" (UniqueName: \"kubernetes.io/projected/655f67e1-9da5-4ccf-b57c-c085d8bb523e-kube-api-access-bd4jp\") pod \"payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv\" (UID: \"655f67e1-9da5-4ccf-b57c-c085d8bb523e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"
Sep 16 18:36:55.627721 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.627459    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f4dc05-42f8-4fc7-8543-15cefc4e80c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw\" (UID: \"22f4dc05-42f8-4fc7-8543-15cefc4e80c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"
Sep 16 18:36:55.628149 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.628130    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655f67e1-9da5-4ccf-b57c-c085d8bb523e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv\" (UID: \"655f67e1-9da5-4ccf-b57c-c085d8bb523e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"
Sep 16 18:36:55.628303 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.628281    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f4dc05-42f8-4fc7-8543-15cefc4e80c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw\" (UID: \"22f4dc05-42f8-4fc7-8543-15cefc4e80c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"
Sep 16 18:36:55.636945 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.636926    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kszh\" (UniqueName: \"kubernetes.io/projected/22f4dc05-42f8-4fc7-8543-15cefc4e80c7-kube-api-access-2kszh\") pod \"payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw\" (UID: \"22f4dc05-42f8-4fc7-8543-15cefc4e80c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"
Sep 16 18:36:55.637026 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.637004    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd4jp\" (UniqueName: \"kubernetes.io/projected/655f67e1-9da5-4ccf-b57c-c085d8bb523e-kube-api-access-bd4jp\") pod \"payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv\" (UID: \"655f67e1-9da5-4ccf-b57c-c085d8bb523e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"
Sep 16 18:36:55.678375 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.678348    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"
Sep 16 18:36:55.719610 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.719566    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"
Sep 16 18:36:55.975280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:55.975255    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"]
Sep 16 18:36:55.977149 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:36:55.977125    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f4dc05_42f8_4fc7_8543_15cefc4e80c7.slice/crio-7e7cb6f5cda9b81cbe6835e3693c961b7ac86c292d6a4d809bf116a68d76936a WatchSource:0}: Error finding container 7e7cb6f5cda9b81cbe6835e3693c961b7ac86c292d6a4d809bf116a68d76936a: Status 404 returned error can't find the container with id 7e7cb6f5cda9b81cbe6835e3693c961b7ac86c292d6a4d809bf116a68d76936a
Sep 16 18:36:56.022560 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:56.022537    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"]
Sep 16 18:36:56.026287 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:36:56.026264    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655f67e1_9da5_4ccf_b57c_c085d8bb523e.slice/crio-0d1a3cc830084b56cd32e05f6045bfbe15663eb36a2f351620f3e6bf64dc0206 WatchSource:0}: Error finding container 0d1a3cc830084b56cd32e05f6045bfbe15663eb36a2f351620f3e6bf64dc0206: Status 404 returned error can't find the container with id 0d1a3cc830084b56cd32e05f6045bfbe15663eb36a2f351620f3e6bf64dc0206
Sep 16 18:36:56.137254 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:56.137187    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv" event={"ID":"655f67e1-9da5-4ccf-b57c-c085d8bb523e","Type":"ContainerStarted","Data":"9b331d7f94c07f93eaabedf38127a7d8fdf9ae859ee3708fd96d277990f3710f"}
Sep 16 18:36:56.137432 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:56.137305    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv" event={"ID":"655f67e1-9da5-4ccf-b57c-c085d8bb523e","Type":"ContainerStarted","Data":"0d1a3cc830084b56cd32e05f6045bfbe15663eb36a2f351620f3e6bf64dc0206"}
Sep 16 18:36:56.137432 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:56.137338    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"
Sep 16 18:36:56.139028 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:56.138998    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw" event={"ID":"22f4dc05-42f8-4fc7-8543-15cefc4e80c7","Type":"ContainerStarted","Data":"a5a4a636c76a9b820ea72acbaf2ca4364f670a62dd1096cb7cf82005e947ecca"}
Sep 16 18:36:56.139148 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:56.139049    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw" event={"ID":"22f4dc05-42f8-4fc7-8543-15cefc4e80c7","Type":"ContainerStarted","Data":"7e7cb6f5cda9b81cbe6835e3693c961b7ac86c292d6a4d809bf116a68d76936a"}
Sep 16 18:36:56.139148 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:56.139074    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"
Sep 16 18:36:56.159622 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:56.159562    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv" podStartSLOduration=1.159546685 podStartE2EDuration="1.159546685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:36:56.157632755 +0000 UTC m=+1239.716800589" watchObservedRunningTime="2026-09-16 18:36:56.159546685 +0000 UTC m=+1239.718714519"
Sep 16 18:36:56.179446 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:36:56.179378    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw" podStartSLOduration=1.179358711 podStartE2EDuration="1.179358711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:36:56.178339846 +0000 UTC m=+1239.737507682" watchObservedRunningTime="2026-09-16 18:36:56.179358711 +0000 UTC m=+1239.738526543"
Sep 16 18:37:00.622137 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:00.622104    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:37:07.154161 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:07.154133    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"
Sep 16 18:37:07.154615 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:07.154202    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"
Sep 16 18:37:17.410388 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.410353    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"]
Sep 16 18:37:17.410866 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.410551    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p" podUID="8e2e1faf-016d-4bfb-be40-85eb52662ca0" containerName="maas-api" containerID="cri-o://10fb28e142ff79f6ecb396efa35e493da2e112399ea52d896e30969ee2f7b33c" gracePeriod=30
Sep 16 18:37:17.512480 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.512431    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"]
Sep 16 18:37:17.512890 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.512851    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz" podUID="c6db1ae7-cf0b-4760-b50c-a1733be0e781" containerName="payload-processing" containerID="cri-o://e5cc9d0ad716ba50d19e55bc725b8377f826e7f5d8045f1c3e581eadaf6e3ab7" gracePeriod=30
Sep 16 18:37:17.554252 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.552257    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"]
Sep 16 18:37:17.554252 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.552561    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7" podUID="afd01819-79e8-4316-83db-dda60ac5aeb3" containerName="payload-pre-processing" containerID="cri-o://b9dac7c897371e4c1e0a9d412f95df6242d81448ebe837a21ec26925ddddf2b5" gracePeriod=30
Sep 16 18:37:17.763115 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.763078    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:37:17.862627 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.862342    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"
Sep 16 18:37:17.865404 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.865384    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"
Sep 16 18:37:17.868648 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.868632    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e2e1faf-016d-4bfb-be40-85eb52662ca0-maas-api-tls\") pod \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\" (UID: \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\") "
Sep 16 18:37:17.868726 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.868663    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2e1faf-016d-4bfb-be40-85eb52662ca0-openshift-service-ca\") pod \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\" (UID: \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\") "
Sep 16 18:37:17.868777 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.868768    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e2e1faf-016d-4bfb-be40-85eb52662ca0-metrics-certs\") pod \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\" (UID: \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\") "
Sep 16 18:37:17.868851 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.868837    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plg5j\" (UniqueName: \"kubernetes.io/projected/8e2e1faf-016d-4bfb-be40-85eb52662ca0-kube-api-access-plg5j\") pod \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\" (UID: \"8e2e1faf-016d-4bfb-be40-85eb52662ca0\") "
Sep 16 18:37:17.869009 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.868987    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2e1faf-016d-4bfb-be40-85eb52662ca0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e2e1faf-016d-4bfb-be40-85eb52662ca0" (UID: "8e2e1faf-016d-4bfb-be40-85eb52662ca0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:37:17.869171 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.869157    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2e1faf-016d-4bfb-be40-85eb52662ca0-openshift-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:17.870883 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.870858    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e2e1faf-016d-4bfb-be40-85eb52662ca0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e2e1faf-016d-4bfb-be40-85eb52662ca0" (UID: "8e2e1faf-016d-4bfb-be40-85eb52662ca0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:37:17.870982 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.870898    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e2e1faf-016d-4bfb-be40-85eb52662ca0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8e2e1faf-016d-4bfb-be40-85eb52662ca0" (UID: "8e2e1faf-016d-4bfb-be40-85eb52662ca0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:37:17.870982 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.870918    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2e1faf-016d-4bfb-be40-85eb52662ca0-kube-api-access-plg5j" (OuterVolumeSpecName: "kube-api-access-plg5j") pod "8e2e1faf-016d-4bfb-be40-85eb52662ca0" (UID: "8e2e1faf-016d-4bfb-be40-85eb52662ca0"). InnerVolumeSpecName "kube-api-access-plg5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:37:17.969920 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.969833    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd696\" (UniqueName: \"kubernetes.io/projected/c6db1ae7-cf0b-4760-b50c-a1733be0e781-kube-api-access-rd696\") pod \"c6db1ae7-cf0b-4760-b50c-a1733be0e781\" (UID: \"c6db1ae7-cf0b-4760-b50c-a1733be0e781\") "
Sep 16 18:37:17.969920 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.969896    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5948h\" (UniqueName: \"kubernetes.io/projected/afd01819-79e8-4316-83db-dda60ac5aeb3-kube-api-access-5948h\") pod \"afd01819-79e8-4316-83db-dda60ac5aeb3\" (UID: \"afd01819-79e8-4316-83db-dda60ac5aeb3\") "
Sep 16 18:37:17.970139 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.969990    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6db1ae7-cf0b-4760-b50c-a1733be0e781-plugins-config-volume\") pod \"c6db1ae7-cf0b-4760-b50c-a1733be0e781\" (UID: \"c6db1ae7-cf0b-4760-b50c-a1733be0e781\") "
Sep 16 18:37:17.970139 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.970034    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd01819-79e8-4316-83db-dda60ac5aeb3-plugins-config-volume\") pod \"afd01819-79e8-4316-83db-dda60ac5aeb3\" (UID: \"afd01819-79e8-4316-83db-dda60ac5aeb3\") "
Sep 16 18:37:17.970416 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.970390    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6db1ae7-cf0b-4760-b50c-a1733be0e781-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c6db1ae7-cf0b-4760-b50c-a1733be0e781" (UID: "c6db1ae7-cf0b-4760-b50c-a1733be0e781"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:37:17.970416 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.970408    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e2e1faf-016d-4bfb-be40-85eb52662ca0-metrics-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:17.970538 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.970416    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd01819-79e8-4316-83db-dda60ac5aeb3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "afd01819-79e8-4316-83db-dda60ac5aeb3" (UID: "afd01819-79e8-4316-83db-dda60ac5aeb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:37:17.970538 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.970428    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plg5j\" (UniqueName: \"kubernetes.io/projected/8e2e1faf-016d-4bfb-be40-85eb52662ca0-kube-api-access-plg5j\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:17.970538 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.970461    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e2e1faf-016d-4bfb-be40-85eb52662ca0-maas-api-tls\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:17.972613 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.972592    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd01819-79e8-4316-83db-dda60ac5aeb3-kube-api-access-5948h" (OuterVolumeSpecName: "kube-api-access-5948h") pod "afd01819-79e8-4316-83db-dda60ac5aeb3" (UID: "afd01819-79e8-4316-83db-dda60ac5aeb3"). InnerVolumeSpecName "kube-api-access-5948h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:37:17.972688 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:17.972617    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6db1ae7-cf0b-4760-b50c-a1733be0e781-kube-api-access-rd696" (OuterVolumeSpecName: "kube-api-access-rd696") pod "c6db1ae7-cf0b-4760-b50c-a1733be0e781" (UID: "c6db1ae7-cf0b-4760-b50c-a1733be0e781"). InnerVolumeSpecName "kube-api-access-rd696". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:37:18.071145 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.071101    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5948h\" (UniqueName: \"kubernetes.io/projected/afd01819-79e8-4316-83db-dda60ac5aeb3-kube-api-access-5948h\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:18.071145 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.071141    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6db1ae7-cf0b-4760-b50c-a1733be0e781-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:18.071145 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.071152    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd01819-79e8-4316-83db-dda60ac5aeb3-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:18.071412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.071163    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd696\" (UniqueName: \"kubernetes.io/projected/c6db1ae7-cf0b-4760-b50c-a1733be0e781-kube-api-access-rd696\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:18.292156 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.292126    2554 generic.go:356] "Generic (PLEG): container finished" podID="afd01819-79e8-4316-83db-dda60ac5aeb3" containerID="b9dac7c897371e4c1e0a9d412f95df6242d81448ebe837a21ec26925ddddf2b5" exitCode=0
Sep 16 18:37:18.292370 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.292171    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"
Sep 16 18:37:18.292370 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.292213    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7" event={"ID":"afd01819-79e8-4316-83db-dda60ac5aeb3","Type":"ContainerDied","Data":"b9dac7c897371e4c1e0a9d412f95df6242d81448ebe837a21ec26925ddddf2b5"}
Sep 16 18:37:18.292370 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.292277    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7" event={"ID":"afd01819-79e8-4316-83db-dda60ac5aeb3","Type":"ContainerDied","Data":"d795bd7999829a64ba48d3e54346130232610d2b81d5ce43969407bec9938db2"}
Sep 16 18:37:18.292370 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.292307    2554 scope.go:117] "RemoveContainer" containerID="b9dac7c897371e4c1e0a9d412f95df6242d81448ebe837a21ec26925ddddf2b5"
Sep 16 18:37:18.293838 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.293814    2554 generic.go:356] "Generic (PLEG): container finished" podID="c6db1ae7-cf0b-4760-b50c-a1733be0e781" containerID="e5cc9d0ad716ba50d19e55bc725b8377f826e7f5d8045f1c3e581eadaf6e3ab7" exitCode=0
Sep 16 18:37:18.293918 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.293847    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"
Sep 16 18:37:18.293918 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.293897    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz" event={"ID":"c6db1ae7-cf0b-4760-b50c-a1733be0e781","Type":"ContainerDied","Data":"e5cc9d0ad716ba50d19e55bc725b8377f826e7f5d8045f1c3e581eadaf6e3ab7"}
Sep 16 18:37:18.294043 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.293949    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz" event={"ID":"c6db1ae7-cf0b-4760-b50c-a1733be0e781","Type":"ContainerDied","Data":"6157bec46fc6309cfdb3cad869abf65bdf605a56d0b73d21f149dce66c634027"}
Sep 16 18:37:18.295501 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.295404    2554 generic.go:356] "Generic (PLEG): container finished" podID="8e2e1faf-016d-4bfb-be40-85eb52662ca0" containerID="10fb28e142ff79f6ecb396efa35e493da2e112399ea52d896e30969ee2f7b33c" exitCode=0
Sep 16 18:37:18.295501 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.295429    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p" event={"ID":"8e2e1faf-016d-4bfb-be40-85eb52662ca0","Type":"ContainerDied","Data":"10fb28e142ff79f6ecb396efa35e493da2e112399ea52d896e30969ee2f7b33c"}
Sep 16 18:37:18.295501 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.295480    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p" event={"ID":"8e2e1faf-016d-4bfb-be40-85eb52662ca0","Type":"ContainerDied","Data":"ec4cf3304b37d26ce893c6396e519478c001d2e9c6188be1869179d9576b6ac1"}
Sep 16 18:37:18.295501 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.295486    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"
Sep 16 18:37:18.307251 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.307213    2554 scope.go:117] "RemoveContainer" containerID="b9dac7c897371e4c1e0a9d412f95df6242d81448ebe837a21ec26925ddddf2b5"
Sep 16 18:37:18.307856 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:37:18.307700    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9dac7c897371e4c1e0a9d412f95df6242d81448ebe837a21ec26925ddddf2b5\": container with ID starting with b9dac7c897371e4c1e0a9d412f95df6242d81448ebe837a21ec26925ddddf2b5 not found: ID does not exist" containerID="b9dac7c897371e4c1e0a9d412f95df6242d81448ebe837a21ec26925ddddf2b5"
Sep 16 18:37:18.308107 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.307984    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9dac7c897371e4c1e0a9d412f95df6242d81448ebe837a21ec26925ddddf2b5"} err="failed to get container status \"b9dac7c897371e4c1e0a9d412f95df6242d81448ebe837a21ec26925ddddf2b5\": rpc error: code = NotFound desc = could not find container \"b9dac7c897371e4c1e0a9d412f95df6242d81448ebe837a21ec26925ddddf2b5\": container with ID starting with b9dac7c897371e4c1e0a9d412f95df6242d81448ebe837a21ec26925ddddf2b5 not found: ID does not exist"
Sep 16 18:37:18.308107 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.308020    2554 scope.go:117] "RemoveContainer" containerID="e5cc9d0ad716ba50d19e55bc725b8377f826e7f5d8045f1c3e581eadaf6e3ab7"
Sep 16 18:37:18.320081 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.320060    2554 scope.go:117] "RemoveContainer" containerID="e5cc9d0ad716ba50d19e55bc725b8377f826e7f5d8045f1c3e581eadaf6e3ab7"
Sep 16 18:37:18.320523 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:37:18.320501    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5cc9d0ad716ba50d19e55bc725b8377f826e7f5d8045f1c3e581eadaf6e3ab7\": container with ID starting with e5cc9d0ad716ba50d19e55bc725b8377f826e7f5d8045f1c3e581eadaf6e3ab7 not found: ID does not exist" containerID="e5cc9d0ad716ba50d19e55bc725b8377f826e7f5d8045f1c3e581eadaf6e3ab7"
Sep 16 18:37:18.320585 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.320531    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5cc9d0ad716ba50d19e55bc725b8377f826e7f5d8045f1c3e581eadaf6e3ab7"} err="failed to get container status \"e5cc9d0ad716ba50d19e55bc725b8377f826e7f5d8045f1c3e581eadaf6e3ab7\": rpc error: code = NotFound desc = could not find container \"e5cc9d0ad716ba50d19e55bc725b8377f826e7f5d8045f1c3e581eadaf6e3ab7\": container with ID starting with e5cc9d0ad716ba50d19e55bc725b8377f826e7f5d8045f1c3e581eadaf6e3ab7 not found: ID does not exist"
Sep 16 18:37:18.320585 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.320550    2554 scope.go:117] "RemoveContainer" containerID="10fb28e142ff79f6ecb396efa35e493da2e112399ea52d896e30969ee2f7b33c"
Sep 16 18:37:18.330498 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.330476    2554 scope.go:117] "RemoveContainer" containerID="10fb28e142ff79f6ecb396efa35e493da2e112399ea52d896e30969ee2f7b33c"
Sep 16 18:37:18.330593 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.330482    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"]
Sep 16 18:37:18.330819 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:37:18.330795    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10fb28e142ff79f6ecb396efa35e493da2e112399ea52d896e30969ee2f7b33c\": container with ID starting with 10fb28e142ff79f6ecb396efa35e493da2e112399ea52d896e30969ee2f7b33c not found: ID does not exist" containerID="10fb28e142ff79f6ecb396efa35e493da2e112399ea52d896e30969ee2f7b33c"
Sep 16 18:37:18.330946 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.330826    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10fb28e142ff79f6ecb396efa35e493da2e112399ea52d896e30969ee2f7b33c"} err="failed to get container status \"10fb28e142ff79f6ecb396efa35e493da2e112399ea52d896e30969ee2f7b33c\": rpc error: code = NotFound desc = could not find container \"10fb28e142ff79f6ecb396efa35e493da2e112399ea52d896e30969ee2f7b33c\": container with ID starting with 10fb28e142ff79f6ecb396efa35e493da2e112399ea52d896e30969ee2f7b33c not found: ID does not exist"
Sep 16 18:37:18.334117 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.334094    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ca9c2-89f45dd7d-t8gv7"]
Sep 16 18:37:18.350546 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.350511    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"]
Sep 16 18:37:18.354815 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.354791    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ca9c2-7c678ffbc7-pwtnz"]
Sep 16 18:37:18.372588 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.372556    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"]
Sep 16 18:37:18.377001 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.376973    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca9c2-6669756678-8fb6p"]
Sep 16 18:37:18.885737 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.885702    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e2e1faf-016d-4bfb-be40-85eb52662ca0" path="/var/lib/kubelet/pods/8e2e1faf-016d-4bfb-be40-85eb52662ca0/volumes"
Sep 16 18:37:18.886651 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.886302    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afd01819-79e8-4316-83db-dda60ac5aeb3" path="/var/lib/kubelet/pods/afd01819-79e8-4316-83db-dda60ac5aeb3/volumes"
Sep 16 18:37:18.886823 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:18.886798    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6db1ae7-cf0b-4760-b50c-a1733be0e781" path="/var/lib/kubelet/pods/c6db1ae7-cf0b-4760-b50c-a1733be0e781/volumes"
Sep 16 18:37:21.084585 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.084550    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7706-9300a2"]
Sep 16 18:37:21.085005 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.084993    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e2e1faf-016d-4bfb-be40-85eb52662ca0" containerName="maas-api"
Sep 16 18:37:21.085005 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.085006    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e2e1faf-016d-4bfb-be40-85eb52662ca0" containerName="maas-api"
Sep 16 18:37:21.085094 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.085018    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6db1ae7-cf0b-4760-b50c-a1733be0e781" containerName="payload-processing"
Sep 16 18:37:21.085094 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.085024    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6db1ae7-cf0b-4760-b50c-a1733be0e781" containerName="payload-processing"
Sep 16 18:37:21.085094 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.085041    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd01819-79e8-4316-83db-dda60ac5aeb3" containerName="payload-pre-processing"
Sep 16 18:37:21.085094 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.085047    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd01819-79e8-4316-83db-dda60ac5aeb3" containerName="payload-pre-processing"
Sep 16 18:37:21.085254 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.085104    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e2e1faf-016d-4bfb-be40-85eb52662ca0" containerName="maas-api"
Sep 16 18:37:21.085254 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.085113    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6db1ae7-cf0b-4760-b50c-a1733be0e781" containerName="payload-processing"
Sep 16 18:37:21.085254 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.085122    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd01819-79e8-4316-83db-dda60ac5aeb3" containerName="payload-pre-processing"
Sep 16 18:37:21.089956 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.089936    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-9300a2"
Sep 16 18:37:21.092423 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.092401    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xhq6g\""
Sep 16 18:37:21.099369 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.099344    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-9300a2"]
Sep 16 18:37:21.202025 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.201974    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc6mx\" (UniqueName: \"kubernetes.io/projected/eb9628e0-0b50-4c5d-b4b3-8f707a30173e-kube-api-access-pc6mx\") pod \"test-curl-7706-9300a2\" (UID: \"eb9628e0-0b50-4c5d-b4b3-8f707a30173e\") " pod="opendatahub/test-curl-7706-9300a2"
Sep 16 18:37:21.302827 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.302791    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6mx\" (UniqueName: \"kubernetes.io/projected/eb9628e0-0b50-4c5d-b4b3-8f707a30173e-kube-api-access-pc6mx\") pod \"test-curl-7706-9300a2\" (UID: \"eb9628e0-0b50-4c5d-b4b3-8f707a30173e\") " pod="opendatahub/test-curl-7706-9300a2"
Sep 16 18:37:21.313983 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.313945    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc6mx\" (UniqueName: \"kubernetes.io/projected/eb9628e0-0b50-4c5d-b4b3-8f707a30173e-kube-api-access-pc6mx\") pod \"test-curl-7706-9300a2\" (UID: \"eb9628e0-0b50-4c5d-b4b3-8f707a30173e\") " pod="opendatahub/test-curl-7706-9300a2"
Sep 16 18:37:21.401812 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.401722    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-9300a2"
Sep 16 18:37:21.678824 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:21.678798    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7706-9300a2"]
Sep 16 18:37:21.682459 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:37:21.682411    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb9628e0_0b50_4c5d_b4b3_8f707a30173e.slice/crio-c13aa43d24979b3ceb2fe74ac38ca2a340d3e06e09d40d8c97e7528f2ea0f98a WatchSource:0}: Error finding container c13aa43d24979b3ceb2fe74ac38ca2a340d3e06e09d40d8c97e7528f2ea0f98a: Status 404 returned error can't find the container with id c13aa43d24979b3ceb2fe74ac38ca2a340d3e06e09d40d8c97e7528f2ea0f98a
Sep 16 18:37:22.328926 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:22.328890    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-9300a2" event={"ID":"eb9628e0-0b50-4c5d-b4b3-8f707a30173e","Type":"ContainerStarted","Data":"2043b54eaed8b9cfcc7a5560b0704ae4c2c308b7c1768bbcb1206afb6c3ad442"}
Sep 16 18:37:22.328926 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:22.328931    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7706-9300a2" event={"ID":"eb9628e0-0b50-4c5d-b4b3-8f707a30173e","Type":"ContainerStarted","Data":"c13aa43d24979b3ceb2fe74ac38ca2a340d3e06e09d40d8c97e7528f2ea0f98a"}
Sep 16 18:37:22.353539 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:22.353473    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7706-9300a2" podStartSLOduration=1.353452508 podStartE2EDuration="1.353452508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:37:22.353209985 +0000 UTC m=+1265.912377821" watchObservedRunningTime="2026-09-16 18:37:22.353452508 +0000 UTC m=+1265.912620345"
Sep 16 18:37:22.998108 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:22.998073    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7706-9300a2"]
Sep 16 18:37:23.001465 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:23.001435    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7706-9300a2"]
Sep 16 18:37:23.334862 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:37:23.334830    2554 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7706-9300a2" container="test-curl-7706-9300a2"
Sep 16 18:37:24.339778 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:24.339740    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7706-9300a2" podUID="eb9628e0-0b50-4c5d-b4b3-8f707a30173e" containerName="test-curl-7706-9300a2" containerID="cri-o://2043b54eaed8b9cfcc7a5560b0704ae4c2c308b7c1768bbcb1206afb6c3ad442" gracePeriod=2
Sep 16 18:37:24.342268 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:37:24.342217    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7706-9300a2\" is forbidden: User \"system:node:ip-10-0-136-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-38.ec2.internal' and this object" podUID="eb9628e0-0b50-4c5d-b4b3-8f707a30173e" pod="opendatahub/test-curl-7706-9300a2"
Sep 16 18:37:26.534347 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:26.534317    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-9300a2"
Sep 16 18:37:26.657980 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:26.657874    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pc6mx\" (UniqueName: \"kubernetes.io/projected/eb9628e0-0b50-4c5d-b4b3-8f707a30173e-kube-api-access-pc6mx\") pod \"eb9628e0-0b50-4c5d-b4b3-8f707a30173e\" (UID: \"eb9628e0-0b50-4c5d-b4b3-8f707a30173e\") "
Sep 16 18:37:26.660437 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:26.660396    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9628e0-0b50-4c5d-b4b3-8f707a30173e-kube-api-access-pc6mx" (OuterVolumeSpecName: "kube-api-access-pc6mx") pod "eb9628e0-0b50-4c5d-b4b3-8f707a30173e" (UID: "eb9628e0-0b50-4c5d-b4b3-8f707a30173e"). InnerVolumeSpecName "kube-api-access-pc6mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:37:26.759290 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:26.759255    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc6mx\" (UniqueName: \"kubernetes.io/projected/eb9628e0-0b50-4c5d-b4b3-8f707a30173e-kube-api-access-pc6mx\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:26.885062 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:26.885030    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb9628e0-0b50-4c5d-b4b3-8f707a30173e" path="/var/lib/kubelet/pods/eb9628e0-0b50-4c5d-b4b3-8f707a30173e/volumes"
Sep 16 18:37:27.362470 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:27.362433    2554 generic.go:356] "Generic (PLEG): container finished" podID="eb9628e0-0b50-4c5d-b4b3-8f707a30173e" containerID="2043b54eaed8b9cfcc7a5560b0704ae4c2c308b7c1768bbcb1206afb6c3ad442" exitCode=137
Sep 16 18:37:27.362686 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:27.362492    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7706-9300a2"
Sep 16 18:37:27.362686 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:27.362533    2554 scope.go:117] "RemoveContainer" containerID="2043b54eaed8b9cfcc7a5560b0704ae4c2c308b7c1768bbcb1206afb6c3ad442"
Sep 16 18:37:27.371497 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:27.371471    2554 scope.go:117] "RemoveContainer" containerID="2043b54eaed8b9cfcc7a5560b0704ae4c2c308b7c1768bbcb1206afb6c3ad442"
Sep 16 18:37:27.371800 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:37:27.371780    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2043b54eaed8b9cfcc7a5560b0704ae4c2c308b7c1768bbcb1206afb6c3ad442\": container with ID starting with 2043b54eaed8b9cfcc7a5560b0704ae4c2c308b7c1768bbcb1206afb6c3ad442 not found: ID does not exist" containerID="2043b54eaed8b9cfcc7a5560b0704ae4c2c308b7c1768bbcb1206afb6c3ad442"
Sep 16 18:37:27.371869 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:27.371810    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2043b54eaed8b9cfcc7a5560b0704ae4c2c308b7c1768bbcb1206afb6c3ad442"} err="failed to get container status \"2043b54eaed8b9cfcc7a5560b0704ae4c2c308b7c1768bbcb1206afb6c3ad442\": rpc error: code = NotFound desc = could not find container \"2043b54eaed8b9cfcc7a5560b0704ae4c2c308b7c1768bbcb1206afb6c3ad442\": container with ID starting with 2043b54eaed8b9cfcc7a5560b0704ae4c2c308b7c1768bbcb1206afb6c3ad442 not found: ID does not exist"
Sep 16 18:37:31.832648 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:31.832606    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:37:33.836670 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:33.836634    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"]
Sep 16 18:37:33.837414 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:33.837390    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9628e0-0b50-4c5d-b4b3-8f707a30173e" containerName="test-curl-7706-9300a2"
Sep 16 18:37:33.837510 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:33.837418    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9628e0-0b50-4c5d-b4b3-8f707a30173e" containerName="test-curl-7706-9300a2"
Sep 16 18:37:33.837560 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:33.837539    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb9628e0-0b50-4c5d-b4b3-8f707a30173e" containerName="test-curl-7706-9300a2"
Sep 16 18:37:33.842292 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:33.842264    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"
Sep 16 18:37:33.846058 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:33.846004    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a462d953\""
Sep 16 18:37:33.846209 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:33.846188    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a462d953-dockercfg-tnrvz\""
Sep 16 18:37:33.852761 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:33.852733    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"]
Sep 16 18:37:33.937119 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:33.937086    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"]
Sep 16 18:37:33.940881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:33.940860    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"
Sep 16 18:37:33.953737 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:33.953711    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"]
Sep 16 18:37:34.039736 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:34.039697    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxmv\" (UniqueName: \"kubernetes.io/projected/9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0-kube-api-access-tlxmv\") pod \"payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s\" (UID: \"9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0\") " pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"
Sep 16 18:37:34.039736 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:34.039746    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6t2v\" (UniqueName: \"kubernetes.io/projected/ab5fcc9a-d49b-4c58-afc6-f1c74be355d1-kube-api-access-n6t2v\") pod \"payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf\" (UID: \"ab5fcc9a-d49b-4c58-afc6-f1c74be355d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"
Sep 16 18:37:34.040015 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:34.039854    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0-plugins-config-volume\") pod \"payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s\" (UID: \"9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0\") " pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"
Sep 16 18:37:34.040015 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:34.039904    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5fcc9a-d49b-4c58-afc6-f1c74be355d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf\" (UID: \"ab5fcc9a-d49b-4c58-afc6-f1c74be355d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"
Sep 16 18:37:34.141288 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:34.141176    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6t2v\" (UniqueName: \"kubernetes.io/projected/ab5fcc9a-d49b-4c58-afc6-f1c74be355d1-kube-api-access-n6t2v\") pod \"payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf\" (UID: \"ab5fcc9a-d49b-4c58-afc6-f1c74be355d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"
Sep 16 18:37:34.141288 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:34.141266    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0-plugins-config-volume\") pod \"payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s\" (UID: \"9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0\") " pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"
Sep 16 18:37:34.141539 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:34.141309    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5fcc9a-d49b-4c58-afc6-f1c74be355d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf\" (UID: \"ab5fcc9a-d49b-4c58-afc6-f1c74be355d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"
Sep 16 18:37:34.141539 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:34.141414    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxmv\" (UniqueName: \"kubernetes.io/projected/9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0-kube-api-access-tlxmv\") pod \"payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s\" (UID: \"9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0\") " pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"
Sep 16 18:37:34.142417 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:34.142390    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0-plugins-config-volume\") pod \"payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s\" (UID: \"9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0\") " pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"
Sep 16 18:37:34.142573 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:34.142549    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5fcc9a-d49b-4c58-afc6-f1c74be355d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf\" (UID: \"ab5fcc9a-d49b-4c58-afc6-f1c74be355d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"
Sep 16 18:37:34.152250 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:34.152208    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxmv\" (UniqueName: \"kubernetes.io/projected/9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0-kube-api-access-tlxmv\") pod \"payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s\" (UID: \"9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0\") " pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"
Sep 16 18:37:34.152754 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:34.152735    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6t2v\" (UniqueName: \"kubernetes.io/projected/ab5fcc9a-d49b-4c58-afc6-f1c74be355d1-kube-api-access-n6t2v\") pod \"payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf\" (UID: \"ab5fcc9a-d49b-4c58-afc6-f1c74be355d1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"
Sep 16 18:37:34.155526 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:34.155508    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"
Sep 16 18:37:34.253373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:34.253331    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"
Sep 16 18:37:34.449350 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:34.449324    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"]
Sep 16 18:37:34.451098 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:37:34.451070    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0dfcd1_df6d_46ce_b36a_78fbb54e48e0.slice/crio-b17967eac0b97850a9dae8cf501c1d96d95d354af24a8289057b3d26dfcd0b6f WatchSource:0}: Error finding container b17967eac0b97850a9dae8cf501c1d96d95d354af24a8289057b3d26dfcd0b6f: Status 404 returned error can't find the container with id b17967eac0b97850a9dae8cf501c1d96d95d354af24a8289057b3d26dfcd0b6f
Sep 16 18:37:34.545796 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:34.545763    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"]
Sep 16 18:37:34.551784 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:37:34.551748    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5fcc9a_d49b_4c58_afc6_f1c74be355d1.slice/crio-a1708acec8f4af59887a96e88585466e9f2bf1a6abd1c6aabf8e460f246dc7e6 WatchSource:0}: Error finding container a1708acec8f4af59887a96e88585466e9f2bf1a6abd1c6aabf8e460f246dc7e6: Status 404 returned error can't find the container with id a1708acec8f4af59887a96e88585466e9f2bf1a6abd1c6aabf8e460f246dc7e6
Sep 16 18:37:35.410582 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:35.410542    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s" event={"ID":"9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0","Type":"ContainerStarted","Data":"12600f2bb9641c3fd66fe68030117ce8907a62150f3897c4f91c137ee6dfe2f5"}
Sep 16 18:37:35.410991 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:35.410734    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"
Sep 16 18:37:35.410991 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:35.410778    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s" event={"ID":"9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0","Type":"ContainerStarted","Data":"b17967eac0b97850a9dae8cf501c1d96d95d354af24a8289057b3d26dfcd0b6f"}
Sep 16 18:37:35.412206 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:35.412182    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf" event={"ID":"ab5fcc9a-d49b-4c58-afc6-f1c74be355d1","Type":"ContainerStarted","Data":"f41b2fb6e3955cd4af926933f10c9141d9cd7a8097862f5c0daf140c56bc6e13"}
Sep 16 18:37:35.412292 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:35.412226    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"
Sep 16 18:37:35.412292 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:35.412275    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf" event={"ID":"ab5fcc9a-d49b-4c58-afc6-f1c74be355d1","Type":"ContainerStarted","Data":"a1708acec8f4af59887a96e88585466e9f2bf1a6abd1c6aabf8e460f246dc7e6"}
Sep 16 18:37:35.448580 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:35.448524    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s" podStartSLOduration=2.448509275 podStartE2EDuration="2.448509275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:37:35.447023506 +0000 UTC m=+1279.006191342" watchObservedRunningTime="2026-09-16 18:37:35.448509275 +0000 UTC m=+1279.007677106"
Sep 16 18:37:35.485489 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:35.485428    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf" podStartSLOduration=2.485404618 podStartE2EDuration="2.485404618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:37:35.483164822 +0000 UTC m=+1279.042332653" watchObservedRunningTime="2026-09-16 18:37:35.485404618 +0000 UTC m=+1279.044572457"
Sep 16 18:37:46.420436 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:46.420402    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"
Sep 16 18:37:46.420915 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:46.420480    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"
Sep 16 18:37:51.084818 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:51.084775    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"]
Sep 16 18:37:51.085353 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:51.084985    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf" podUID="1f794beb-3530-46d7-a892-b5e59344638c" containerName="istio-proxy" containerID="cri-o://e90183f4c31856a36e9c953241db9e10c3c305224907ca4bfd5ff153e835327f" gracePeriod=30
Sep 16 18:37:56.359372 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.359296    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:37:56.468194 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468153    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-workload-socket\") pod \"1f794beb-3530-46d7-a892-b5e59344638c\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") "
Sep 16 18:37:56.468398 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468215    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-istio-data\") pod \"1f794beb-3530-46d7-a892-b5e59344638c\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") "
Sep 16 18:37:56.468398 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468269    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f794beb-3530-46d7-a892-b5e59344638c-istio-podinfo\") pod \"1f794beb-3530-46d7-a892-b5e59344638c\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") "
Sep 16 18:37:56.468398 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468298    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-workload-certs\") pod \"1f794beb-3530-46d7-a892-b5e59344638c\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") "
Sep 16 18:37:56.468398 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468336    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-istio-envoy\") pod \"1f794beb-3530-46d7-a892-b5e59344638c\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") "
Sep 16 18:37:56.468398 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468379    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-credential-socket\") pod \"1f794beb-3530-46d7-a892-b5e59344638c\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") "
Sep 16 18:37:56.468701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468406    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f794beb-3530-46d7-a892-b5e59344638c-istio-token\") pod \"1f794beb-3530-46d7-a892-b5e59344638c\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") "
Sep 16 18:37:56.468701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468461    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f794beb-3530-46d7-a892-b5e59344638c-istiod-ca-cert\") pod \"1f794beb-3530-46d7-a892-b5e59344638c\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") "
Sep 16 18:37:56.468701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468471    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1f794beb-3530-46d7-a892-b5e59344638c" (UID: "1f794beb-3530-46d7-a892-b5e59344638c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:56.468701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468485    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-istio-data" (OuterVolumeSpecName: "istio-data") pod "1f794beb-3530-46d7-a892-b5e59344638c" (UID: "1f794beb-3530-46d7-a892-b5e59344638c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:56.468701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468508    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzgxz\" (UniqueName: \"kubernetes.io/projected/1f794beb-3530-46d7-a892-b5e59344638c-kube-api-access-zzgxz\") pod \"1f794beb-3530-46d7-a892-b5e59344638c\" (UID: \"1f794beb-3530-46d7-a892-b5e59344638c\") "
Sep 16 18:37:56.468701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468692    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1f794beb-3530-46d7-a892-b5e59344638c" (UID: "1f794beb-3530-46d7-a892-b5e59344638c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:56.468988 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468820    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f794beb-3530-46d7-a892-b5e59344638c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1f794beb-3530-46d7-a892-b5e59344638c" (UID: "1f794beb-3530-46d7-a892-b5e59344638c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:37:56.468988 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468877    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1f794beb-3530-46d7-a892-b5e59344638c" (UID: "1f794beb-3530-46d7-a892-b5e59344638c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:56.468988 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468862    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:56.468988 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468939    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:56.468988 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.468949    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:56.471083 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.471055    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f794beb-3530-46d7-a892-b5e59344638c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1f794beb-3530-46d7-a892-b5e59344638c" (UID: "1f794beb-3530-46d7-a892-b5e59344638c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:37:56.471202 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.471088    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1f794beb-3530-46d7-a892-b5e59344638c" (UID: "1f794beb-3530-46d7-a892-b5e59344638c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:56.471202 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.471118    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f794beb-3530-46d7-a892-b5e59344638c-kube-api-access-zzgxz" (OuterVolumeSpecName: "kube-api-access-zzgxz") pod "1f794beb-3530-46d7-a892-b5e59344638c" (UID: "1f794beb-3530-46d7-a892-b5e59344638c"). InnerVolumeSpecName "kube-api-access-zzgxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:37:56.471202 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.471117    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f794beb-3530-46d7-a892-b5e59344638c-istio-token" (OuterVolumeSpecName: "istio-token") pod "1f794beb-3530-46d7-a892-b5e59344638c" (UID: "1f794beb-3530-46d7-a892-b5e59344638c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:37:56.542713 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.542677    2554 generic.go:356] "Generic (PLEG): container finished" podID="1f794beb-3530-46d7-a892-b5e59344638c" containerID="e90183f4c31856a36e9c953241db9e10c3c305224907ca4bfd5ff153e835327f" exitCode=0
Sep 16 18:37:56.542923 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.542738    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"
Sep 16 18:37:56.542923 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.542764    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf" event={"ID":"1f794beb-3530-46d7-a892-b5e59344638c","Type":"ContainerDied","Data":"e90183f4c31856a36e9c953241db9e10c3c305224907ca4bfd5ff153e835327f"}
Sep 16 18:37:56.542923 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.542813    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf" event={"ID":"1f794beb-3530-46d7-a892-b5e59344638c","Type":"ContainerDied","Data":"138dedd61560833fee6a2999f574b4879081759e874f978b4b611ffa8532d46c"}
Sep 16 18:37:56.542923 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.542843    2554 scope.go:117] "RemoveContainer" containerID="e90183f4c31856a36e9c953241db9e10c3c305224907ca4bfd5ff153e835327f"
Sep 16 18:37:56.552928 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.552904    2554 scope.go:117] "RemoveContainer" containerID="bfb3cf25e2cdd2bbe0e5c105577f1cbb57db5965bfa6b69d5d4f7f813fb0ed60"
Sep 16 18:37:56.561115 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.561097    2554 scope.go:117] "RemoveContainer" containerID="e90183f4c31856a36e9c953241db9e10c3c305224907ca4bfd5ff153e835327f"
Sep 16 18:37:56.561433 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:37:56.561415    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e90183f4c31856a36e9c953241db9e10c3c305224907ca4bfd5ff153e835327f\": container with ID starting with e90183f4c31856a36e9c953241db9e10c3c305224907ca4bfd5ff153e835327f not found: ID does not exist" containerID="e90183f4c31856a36e9c953241db9e10c3c305224907ca4bfd5ff153e835327f"
Sep 16 18:37:56.561490 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.561442    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e90183f4c31856a36e9c953241db9e10c3c305224907ca4bfd5ff153e835327f"} err="failed to get container status \"e90183f4c31856a36e9c953241db9e10c3c305224907ca4bfd5ff153e835327f\": rpc error: code = NotFound desc = could not find container \"e90183f4c31856a36e9c953241db9e10c3c305224907ca4bfd5ff153e835327f\": container with ID starting with e90183f4c31856a36e9c953241db9e10c3c305224907ca4bfd5ff153e835327f not found: ID does not exist"
Sep 16 18:37:56.561490 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.561465    2554 scope.go:117] "RemoveContainer" containerID="bfb3cf25e2cdd2bbe0e5c105577f1cbb57db5965bfa6b69d5d4f7f813fb0ed60"
Sep 16 18:37:56.561745 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:37:56.561724    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb3cf25e2cdd2bbe0e5c105577f1cbb57db5965bfa6b69d5d4f7f813fb0ed60\": container with ID starting with bfb3cf25e2cdd2bbe0e5c105577f1cbb57db5965bfa6b69d5d4f7f813fb0ed60 not found: ID does not exist" containerID="bfb3cf25e2cdd2bbe0e5c105577f1cbb57db5965bfa6b69d5d4f7f813fb0ed60"
Sep 16 18:37:56.561793 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.561751    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb3cf25e2cdd2bbe0e5c105577f1cbb57db5965bfa6b69d5d4f7f813fb0ed60"} err="failed to get container status \"bfb3cf25e2cdd2bbe0e5c105577f1cbb57db5965bfa6b69d5d4f7f813fb0ed60\": rpc error: code = NotFound desc = could not find container \"bfb3cf25e2cdd2bbe0e5c105577f1cbb57db5965bfa6b69d5d4f7f813fb0ed60\": container with ID starting with bfb3cf25e2cdd2bbe0e5c105577f1cbb57db5965bfa6b69d5d4f7f813fb0ed60 not found: ID does not exist"
Sep 16 18:37:56.569749 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.569726    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f794beb-3530-46d7-a892-b5e59344638c-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:56.569819 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.569750    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f794beb-3530-46d7-a892-b5e59344638c-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:56.569819 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.569760    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzgxz\" (UniqueName: \"kubernetes.io/projected/1f794beb-3530-46d7-a892-b5e59344638c-kube-api-access-zzgxz\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:56.569819 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.569769    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f794beb-3530-46d7-a892-b5e59344638c-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:56.569819 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.569777    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:56.569819 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.569784    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f794beb-3530-46d7-a892-b5e59344638c-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:37:56.584438 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.584409    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"]
Sep 16 18:37:56.593196 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.593164    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca9c2-openshift-default-6bd6f8d54b-mvjpf"]
Sep 16 18:37:56.885109 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:56.885075    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f794beb-3530-46d7-a892-b5e59344638c" path="/var/lib/kubelet/pods/1f794beb-3530-46d7-a892-b5e59344638c/volumes"
Sep 16 18:37:58.698927 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.698892    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"]
Sep 16 18:37:58.699408 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.699393    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f794beb-3530-46d7-a892-b5e59344638c" containerName="istio-proxy"
Sep 16 18:37:58.699460 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.699409    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f794beb-3530-46d7-a892-b5e59344638c" containerName="istio-proxy"
Sep 16 18:37:58.699495 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.699487    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f794beb-3530-46d7-a892-b5e59344638c" containerName="istio-proxy"
Sep 16 18:37:58.699529 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.699497    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f794beb-3530-46d7-a892-b5e59344638c" containerName="istio-proxy"
Sep 16 18:37:58.704177 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.704156    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.707688 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.707669    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a24d09\"/\"default-dockercfg-m4w58\""
Sep 16 18:37:58.707810 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.707684    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a24d09\"/\"kube-root-ca.crt\""
Sep 16 18:37:58.707810 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.707671    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a24d09\"/\"distinct-2-w0-a24d09-kserve-self-signed-certs\""
Sep 16 18:37:58.707810 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.707740    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a24d09\"/\"openshift-service-ca.crt\""
Sep 16 18:37:58.715117 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.715096    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"]
Sep 16 18:37:58.791661 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.791629    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvcwg\" (UniqueName: \"kubernetes.io/projected/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-kube-api-access-bvcwg\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.791837 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.791674    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-dshm\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.791837 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.791744    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-home\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.791837 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.791785    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-tmp-dir\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.791976 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.791846    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-model-cache\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.791976 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.791875    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-tls-certs\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.893426 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.893390    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-model-cache\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.893426 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.893427    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-tls-certs\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.893665 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.893496    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvcwg\" (UniqueName: \"kubernetes.io/projected/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-kube-api-access-bvcwg\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.893665 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.893525    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-dshm\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.893665 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.893558    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-home\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.893665 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.893577    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-tmp-dir\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.893878 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.893837    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-model-cache\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.893929 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.893881    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-home\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.893977 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.893937    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-tmp-dir\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.895751 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.895725    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-dshm\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.896017 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.896001    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-tls-certs\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:58.901737 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:58.901718    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvcwg\" (UniqueName: \"kubernetes.io/projected/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-kube-api-access-bvcwg\") pod \"distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") " pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:59.013018 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:59.012975    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:37:59.298371 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:59.298289    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"]
Sep 16 18:37:59.301530 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:37:59.301500    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbee55eb5_c320_4715_b2a9_6c8b6ed4d800.slice/crio-3a2a395ff2390a4f45a3a56bf30a07b146133e181231c8b4053eb1c5f7235d24 WatchSource:0}: Error finding container 3a2a395ff2390a4f45a3a56bf30a07b146133e181231c8b4053eb1c5f7235d24: Status 404 returned error can't find the container with id 3a2a395ff2390a4f45a3a56bf30a07b146133e181231c8b4053eb1c5f7235d24
Sep 16 18:37:59.565113 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:59.565016    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g" event={"ID":"bee55eb5-c320-4715-b2a9-6c8b6ed4d800","Type":"ContainerStarted","Data":"c31d6978ef8e0258450612fec2413e34f8b7b60a7fc909dbe70fc25f5cdca129"}
Sep 16 18:37:59.565113 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:59.565063    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g" event={"ID":"bee55eb5-c320-4715-b2a9-6c8b6ed4d800","Type":"ContainerStarted","Data":"3a2a395ff2390a4f45a3a56bf30a07b146133e181231c8b4053eb1c5f7235d24"}
Sep 16 18:37:59.588500 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:37:59.588436    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g" podStartSLOduration=1.588420056 podStartE2EDuration="1.588420056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:37:59.588038717 +0000 UTC m=+1303.147206551" watchObservedRunningTime="2026-09-16 18:37:59.588420056 +0000 UTC m=+1303.147587891"
Sep 16 18:38:00.252701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.252655    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:38:00.701583 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.701482    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"]
Sep 16 18:38:00.702033 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.702013    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f794beb-3530-46d7-a892-b5e59344638c" containerName="istio-proxy"
Sep 16 18:38:00.702134 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.702035    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f794beb-3530-46d7-a892-b5e59344638c" containerName="istio-proxy"
Sep 16 18:38:00.705564 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.705540    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:00.709920 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.709900    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-972df0-serving-cert\""
Sep 16 18:38:00.719419 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.719394    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"]
Sep 16 18:38:00.811629 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.811588    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24d554af-5c77-4adc-8ef9-f49e6b184ac2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:00.811629 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.811635    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d554af-5c77-4adc-8ef9-f49e6b184ac2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:00.811902 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.811656    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24d554af-5c77-4adc-8ef9-f49e6b184ac2-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:00.811902 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.811852    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2c5j\" (UniqueName: \"kubernetes.io/projected/24d554af-5c77-4adc-8ef9-f49e6b184ac2-kube-api-access-h2c5j\") pod \"maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:00.912978 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.912937    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2c5j\" (UniqueName: \"kubernetes.io/projected/24d554af-5c77-4adc-8ef9-f49e6b184ac2-kube-api-access-h2c5j\") pod \"maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:00.913166 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.913001    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24d554af-5c77-4adc-8ef9-f49e6b184ac2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:00.913166 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.913038    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d554af-5c77-4adc-8ef9-f49e6b184ac2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:00.913166 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.913057    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24d554af-5c77-4adc-8ef9-f49e6b184ac2-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:00.913338 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:38:00.913160    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-serving-cert: secret "maas-api-e2e-shared-a-w3-972df0-serving-cert" not found
Sep 16 18:38:00.913338 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:38:00.913272    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24d554af-5c77-4adc-8ef9-f49e6b184ac2-maas-api-tls podName:24d554af-5c77-4adc-8ef9-f49e6b184ac2 nodeName:}" failed. No retries permitted until 2026-09-16 18:38:01.413215551 +0000 UTC m=+1304.972383379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24d554af-5c77-4adc-8ef9-f49e6b184ac2-maas-api-tls") pod "maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg" (UID: "24d554af-5c77-4adc-8ef9-f49e6b184ac2") : secret "maas-api-e2e-shared-a-w3-972df0-serving-cert" not found
Sep 16 18:38:00.913829 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.913807    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d554af-5c77-4adc-8ef9-f49e6b184ac2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:00.915582 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.915562    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24d554af-5c77-4adc-8ef9-f49e6b184ac2-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:00.924609 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:00.924587    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2c5j\" (UniqueName: \"kubernetes.io/projected/24d554af-5c77-4adc-8ef9-f49e6b184ac2-kube-api-access-h2c5j\") pod \"maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:01.417978 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:01.417946    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24d554af-5c77-4adc-8ef9-f49e6b184ac2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:01.420908 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:01.420874    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24d554af-5c77-4adc-8ef9-f49e6b184ac2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:01.617318 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:01.617281    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:01.898760 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:01.898733    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"]
Sep 16 18:38:01.900722 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:38:01.900699    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d554af_5c77_4adc_8ef9_f49e6b184ac2.slice/crio-244c9b40a22cabef55becfbfa7934355b63867d45b526a85c5b93e6427625597 WatchSource:0}: Error finding container 244c9b40a22cabef55becfbfa7934355b63867d45b526a85c5b93e6427625597: Status 404 returned error can't find the container with id 244c9b40a22cabef55becfbfa7934355b63867d45b526a85c5b93e6427625597
Sep 16 18:38:01.902138 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:01.902119    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:38:02.584421 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:02.584377    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg" event={"ID":"24d554af-5c77-4adc-8ef9-f49e6b184ac2","Type":"ContainerStarted","Data":"caaba526d877c25213213541a28a60cb7f0a95de0c0901f4600d70ccf8d1954e"}
Sep 16 18:38:02.584421 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:02.584429    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg" event={"ID":"24d554af-5c77-4adc-8ef9-f49e6b184ac2","Type":"ContainerStarted","Data":"244c9b40a22cabef55becfbfa7934355b63867d45b526a85c5b93e6427625597"}
Sep 16 18:38:02.584874 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:02.584465    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:02.611084 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:02.611027    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg" podStartSLOduration=2.117433305 podStartE2EDuration="2.611011294s" totalImagesPullingTime="493.577989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:38:02.608685975 +0000 UTC m=+1306.167853822" watchObservedRunningTime="2026-09-16 18:38:02.611011294 +0000 UTC m=+1306.170179127"
Sep 16 18:38:08.595686 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:08.595654    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:38:09.014089 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:09.014058    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:38:09.014284 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:09.014106    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:38:09.026871 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:09.026845    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:38:09.640886 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:09.640857    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:38:16.975681 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:16.975649    2554 scope.go:117] "RemoveContainer" containerID="9b9a210f9d227b4796ef9703273c42150df26d627664b3e6d88ac80dc8ec571f"
Sep 16 18:38:18.899173 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:18.899143    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"]
Sep 16 18:38:18.904010 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:18.903993    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:18.907607 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:18.907581    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a462d953\"/\"e2e-lifecycle-model-a462d953-kserve-self-signed-certs\""
Sep 16 18:38:18.908349 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:18.908325    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a462d953\"/\"kube-root-ca.crt\""
Sep 16 18:38:18.908465 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:18.908389    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a462d953\"/\"openshift-service-ca.crt\""
Sep 16 18:38:18.912317 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:18.912298    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a462d953\"/\"default-dockercfg-fpc4x\""
Sep 16 18:38:18.916946 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:18.916921    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"]
Sep 16 18:38:18.985280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:18.985214    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4366fe17-ae3c-4824-a12a-5cceed82a91a-tls-certs\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:18.985476 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:18.985347    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-home\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:18.985619 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:18.985599    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-tmp-dir\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:18.985978 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:18.985944    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-model-cache\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:18.986078 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:18.985990    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb2tn\" (UniqueName: \"kubernetes.io/projected/4366fe17-ae3c-4824-a12a-5cceed82a91a-kube-api-access-zb2tn\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:18.986078 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:18.986033    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-dshm\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:19.087001 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.086967    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-model-cache\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:19.087001 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.087003    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb2tn\" (UniqueName: \"kubernetes.io/projected/4366fe17-ae3c-4824-a12a-5cceed82a91a-kube-api-access-zb2tn\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:19.087264 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.087019    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-dshm\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:19.087264 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.087085    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4366fe17-ae3c-4824-a12a-5cceed82a91a-tls-certs\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:19.087264 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.087109    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-home\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:19.087264 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.087179    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-tmp-dir\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:19.087486 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.087411    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-model-cache\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:19.087537 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.087502    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-home\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:19.087575 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.087552    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-tmp-dir\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:19.089439 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.089412    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-dshm\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:19.089637 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.089620    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4366fe17-ae3c-4824-a12a-5cceed82a91a-tls-certs\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:19.097870 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.097838    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb2tn\" (UniqueName: \"kubernetes.io/projected/4366fe17-ae3c-4824-a12a-5cceed82a91a-kube-api-access-zb2tn\") pod \"e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") " pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:19.213905 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.213812    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:19.494304 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.494269    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"]
Sep 16 18:38:19.495897 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:38:19.495872    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4366fe17_ae3c_4824_a12a_5cceed82a91a.slice/crio-a373fb31d7f8cb28e9a965cb5ef8b32808c5571fe435dc82362432aafbba3100 WatchSource:0}: Error finding container a373fb31d7f8cb28e9a965cb5ef8b32808c5571fe435dc82362432aafbba3100: Status 404 returned error can't find the container with id a373fb31d7f8cb28e9a965cb5ef8b32808c5571fe435dc82362432aafbba3100
Sep 16 18:38:19.693412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.693365    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt" event={"ID":"4366fe17-ae3c-4824-a12a-5cceed82a91a","Type":"ContainerStarted","Data":"102573aa8f556c50991f579f27450e4397720cde39a2b98a6dea99b1a51bbbe7"}
Sep 16 18:38:19.693412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.693413    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt" event={"ID":"4366fe17-ae3c-4824-a12a-5cceed82a91a","Type":"ContainerStarted","Data":"a373fb31d7f8cb28e9a965cb5ef8b32808c5571fe435dc82362432aafbba3100"}
Sep 16 18:38:19.713201 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:19.713137    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt" podStartSLOduration=1.713117547 podStartE2EDuration="1.713117547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:38:19.712438138 +0000 UTC m=+1323.271605971" watchObservedRunningTime="2026-09-16 18:38:19.713117547 +0000 UTC m=+1323.272285383"
Sep 16 18:38:28.553292 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:28.553247    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:38:29.214903 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:29.214870    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:29.215078 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:29.214921    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:29.229695 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:29.229666    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:29.768691 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:29.768659    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:38:53.125195 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:38:53.125157    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:39:23.023824 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:23.023785    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:39:44.579913 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:44.579872    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"]
Sep 16 18:39:44.580416 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:44.580103    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s" podUID="9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0" containerName="payload-processing" containerID="cri-o://12600f2bb9641c3fd66fe68030117ce8907a62150f3897c4f91c137ee6dfe2f5" gracePeriod=30
Sep 16 18:39:44.589341 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:44.589309    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"]
Sep 16 18:39:44.589711 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:44.589539    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf" podUID="ab5fcc9a-d49b-4c58-afc6-f1c74be355d1" containerName="payload-pre-processing" containerID="cri-o://f41b2fb6e3955cd4af926933f10c9141d9cd7a8097862f5c0daf140c56bc6e13" gracePeriod=30
Sep 16 18:39:44.873039 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:44.873018    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"
Sep 16 18:39:44.875887 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:44.875869    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"
Sep 16 18:39:44.950930 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:44.950900    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6t2v\" (UniqueName: \"kubernetes.io/projected/ab5fcc9a-d49b-4c58-afc6-f1c74be355d1-kube-api-access-n6t2v\") pod \"ab5fcc9a-d49b-4c58-afc6-f1c74be355d1\" (UID: \"ab5fcc9a-d49b-4c58-afc6-f1c74be355d1\") "
Sep 16 18:39:44.951108 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:44.950957    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0-plugins-config-volume\") pod \"9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0\" (UID: \"9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0\") "
Sep 16 18:39:44.951108 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:44.951014    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5fcc9a-d49b-4c58-afc6-f1c74be355d1-plugins-config-volume\") pod \"ab5fcc9a-d49b-4c58-afc6-f1c74be355d1\" (UID: \"ab5fcc9a-d49b-4c58-afc6-f1c74be355d1\") "
Sep 16 18:39:44.951108 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:44.951077    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlxmv\" (UniqueName: \"kubernetes.io/projected/9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0-kube-api-access-tlxmv\") pod \"9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0\" (UID: \"9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0\") "
Sep 16 18:39:44.951502 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:44.951450    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5fcc9a-d49b-4c58-afc6-f1c74be355d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ab5fcc9a-d49b-4c58-afc6-f1c74be355d1" (UID: "ab5fcc9a-d49b-4c58-afc6-f1c74be355d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:39:44.951502 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:44.951456    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0" (UID: "9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:39:44.953061 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:44.953037    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5fcc9a-d49b-4c58-afc6-f1c74be355d1-kube-api-access-n6t2v" (OuterVolumeSpecName: "kube-api-access-n6t2v") pod "ab5fcc9a-d49b-4c58-afc6-f1c74be355d1" (UID: "ab5fcc9a-d49b-4c58-afc6-f1c74be355d1"). InnerVolumeSpecName "kube-api-access-n6t2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:39:44.953264 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:44.953203    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0-kube-api-access-tlxmv" (OuterVolumeSpecName: "kube-api-access-tlxmv") pod "9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0" (UID: "9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0"). InnerVolumeSpecName "kube-api-access-tlxmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:39:45.052167 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.052136    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlxmv\" (UniqueName: \"kubernetes.io/projected/9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0-kube-api-access-tlxmv\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:39:45.052167 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.052164    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6t2v\" (UniqueName: \"kubernetes.io/projected/ab5fcc9a-d49b-4c58-afc6-f1c74be355d1-kube-api-access-n6t2v\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:39:45.052167 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.052173    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:39:45.052400 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.052182    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5fcc9a-d49b-4c58-afc6-f1c74be355d1-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:39:45.187021 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.186933    2554 generic.go:356] "Generic (PLEG): container finished" podID="ab5fcc9a-d49b-4c58-afc6-f1c74be355d1" containerID="f41b2fb6e3955cd4af926933f10c9141d9cd7a8097862f5c0daf140c56bc6e13" exitCode=0
Sep 16 18:39:45.187021 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.186980    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"
Sep 16 18:39:45.187227 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.187022    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf" event={"ID":"ab5fcc9a-d49b-4c58-afc6-f1c74be355d1","Type":"ContainerDied","Data":"f41b2fb6e3955cd4af926933f10c9141d9cd7a8097862f5c0daf140c56bc6e13"}
Sep 16 18:39:45.187227 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.187074    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf" event={"ID":"ab5fcc9a-d49b-4c58-afc6-f1c74be355d1","Type":"ContainerDied","Data":"a1708acec8f4af59887a96e88585466e9f2bf1a6abd1c6aabf8e460f246dc7e6"}
Sep 16 18:39:45.187227 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.187100    2554 scope.go:117] "RemoveContainer" containerID="f41b2fb6e3955cd4af926933f10c9141d9cd7a8097862f5c0daf140c56bc6e13"
Sep 16 18:39:45.188405 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.188384    2554 generic.go:356] "Generic (PLEG): container finished" podID="9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0" containerID="12600f2bb9641c3fd66fe68030117ce8907a62150f3897c4f91c137ee6dfe2f5" exitCode=0
Sep 16 18:39:45.188546 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.188424    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"
Sep 16 18:39:45.188546 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.188439    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s" event={"ID":"9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0","Type":"ContainerDied","Data":"12600f2bb9641c3fd66fe68030117ce8907a62150f3897c4f91c137ee6dfe2f5"}
Sep 16 18:39:45.188546 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.188479    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s" event={"ID":"9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0","Type":"ContainerDied","Data":"b17967eac0b97850a9dae8cf501c1d96d95d354af24a8289057b3d26dfcd0b6f"}
Sep 16 18:39:45.197472 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.197455    2554 scope.go:117] "RemoveContainer" containerID="f41b2fb6e3955cd4af926933f10c9141d9cd7a8097862f5c0daf140c56bc6e13"
Sep 16 18:39:45.197779 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:39:45.197758    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41b2fb6e3955cd4af926933f10c9141d9cd7a8097862f5c0daf140c56bc6e13\": container with ID starting with f41b2fb6e3955cd4af926933f10c9141d9cd7a8097862f5c0daf140c56bc6e13 not found: ID does not exist" containerID="f41b2fb6e3955cd4af926933f10c9141d9cd7a8097862f5c0daf140c56bc6e13"
Sep 16 18:39:45.197868 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.197804    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41b2fb6e3955cd4af926933f10c9141d9cd7a8097862f5c0daf140c56bc6e13"} err="failed to get container status \"f41b2fb6e3955cd4af926933f10c9141d9cd7a8097862f5c0daf140c56bc6e13\": rpc error: code = NotFound desc = could not find container \"f41b2fb6e3955cd4af926933f10c9141d9cd7a8097862f5c0daf140c56bc6e13\": container with ID starting with f41b2fb6e3955cd4af926933f10c9141d9cd7a8097862f5c0daf140c56bc6e13 not found: ID does not exist"
Sep 16 18:39:45.197868 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.197827    2554 scope.go:117] "RemoveContainer" containerID="12600f2bb9641c3fd66fe68030117ce8907a62150f3897c4f91c137ee6dfe2f5"
Sep 16 18:39:45.205491 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.205470    2554 scope.go:117] "RemoveContainer" containerID="12600f2bb9641c3fd66fe68030117ce8907a62150f3897c4f91c137ee6dfe2f5"
Sep 16 18:39:45.205729 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:39:45.205711    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12600f2bb9641c3fd66fe68030117ce8907a62150f3897c4f91c137ee6dfe2f5\": container with ID starting with 12600f2bb9641c3fd66fe68030117ce8907a62150f3897c4f91c137ee6dfe2f5 not found: ID does not exist" containerID="12600f2bb9641c3fd66fe68030117ce8907a62150f3897c4f91c137ee6dfe2f5"
Sep 16 18:39:45.205791 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.205739    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12600f2bb9641c3fd66fe68030117ce8907a62150f3897c4f91c137ee6dfe2f5"} err="failed to get container status \"12600f2bb9641c3fd66fe68030117ce8907a62150f3897c4f91c137ee6dfe2f5\": rpc error: code = NotFound desc = could not find container \"12600f2bb9641c3fd66fe68030117ce8907a62150f3897c4f91c137ee6dfe2f5\": container with ID starting with 12600f2bb9641c3fd66fe68030117ce8907a62150f3897c4f91c137ee6dfe2f5 not found: ID does not exist"
Sep 16 18:39:45.208898 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.208878    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"]
Sep 16 18:39:45.213085 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.213060    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a462d953-6f968d68f8-wpkmf"]
Sep 16 18:39:45.223162 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.223139    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"]
Sep 16 18:39:45.228766 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:45.228743    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a462d953-6c9db455c-2ht9s"]
Sep 16 18:39:46.884323 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:46.884291    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0" path="/var/lib/kubelet/pods/9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0/volumes"
Sep 16 18:39:46.884724 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:46.884686    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab5fcc9a-d49b-4c58-afc6-f1c74be355d1" path="/var/lib/kubelet/pods/ab5fcc9a-d49b-4c58-afc6-f1c74be355d1/volumes"
Sep 16 18:39:52.844635 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:52.844597    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"]
Sep 16 18:39:53.457395 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:39:53.457355    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:40:07.863766 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:07.863716    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt" podUID="4366fe17-ae3c-4824-a12a-5cceed82a91a" containerName="main" containerID="cri-o://102573aa8f556c50991f579f27450e4397720cde39a2b98a6dea99b1a51bbbe7" gracePeriod=45
Sep 16 18:40:08.123422 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.123353    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:40:08.180304 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.180271    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4366fe17-ae3c-4824-a12a-5cceed82a91a-tls-certs\") pod \"4366fe17-ae3c-4824-a12a-5cceed82a91a\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") "
Sep 16 18:40:08.180463 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.180323    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb2tn\" (UniqueName: \"kubernetes.io/projected/4366fe17-ae3c-4824-a12a-5cceed82a91a-kube-api-access-zb2tn\") pod \"4366fe17-ae3c-4824-a12a-5cceed82a91a\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") "
Sep 16 18:40:08.180463 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.180372    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-model-cache\") pod \"4366fe17-ae3c-4824-a12a-5cceed82a91a\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") "
Sep 16 18:40:08.180574 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.180478    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-tmp-dir\") pod \"4366fe17-ae3c-4824-a12a-5cceed82a91a\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") "
Sep 16 18:40:08.180574 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.180520    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-dshm\") pod \"4366fe17-ae3c-4824-a12a-5cceed82a91a\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") "
Sep 16 18:40:08.180574 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.180545    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-home\") pod \"4366fe17-ae3c-4824-a12a-5cceed82a91a\" (UID: \"4366fe17-ae3c-4824-a12a-5cceed82a91a\") "
Sep 16 18:40:08.180833 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.180755    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-model-cache" (OuterVolumeSpecName: "model-cache") pod "4366fe17-ae3c-4824-a12a-5cceed82a91a" (UID: "4366fe17-ae3c-4824-a12a-5cceed82a91a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:40:08.180833 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.180768    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4366fe17-ae3c-4824-a12a-5cceed82a91a" (UID: "4366fe17-ae3c-4824-a12a-5cceed82a91a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:40:08.181053 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.180970    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-tmp-dir\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:40:08.181053 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.180977    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-home" (OuterVolumeSpecName: "home") pod "4366fe17-ae3c-4824-a12a-5cceed82a91a" (UID: "4366fe17-ae3c-4824-a12a-5cceed82a91a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:40:08.181053 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.180988    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-model-cache\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:40:08.182694 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.182628    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4366fe17-ae3c-4824-a12a-5cceed82a91a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4366fe17-ae3c-4824-a12a-5cceed82a91a" (UID: "4366fe17-ae3c-4824-a12a-5cceed82a91a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:40:08.182975 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.182944    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4366fe17-ae3c-4824-a12a-5cceed82a91a-kube-api-access-zb2tn" (OuterVolumeSpecName: "kube-api-access-zb2tn") pod "4366fe17-ae3c-4824-a12a-5cceed82a91a" (UID: "4366fe17-ae3c-4824-a12a-5cceed82a91a"). InnerVolumeSpecName "kube-api-access-zb2tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:40:08.183070 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.183018    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-dshm" (OuterVolumeSpecName: "dshm") pod "4366fe17-ae3c-4824-a12a-5cceed82a91a" (UID: "4366fe17-ae3c-4824-a12a-5cceed82a91a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:40:08.282292 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.282225    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-dshm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:40:08.282292 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.282286    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4366fe17-ae3c-4824-a12a-5cceed82a91a-home\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:40:08.282292 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.282294    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4366fe17-ae3c-4824-a12a-5cceed82a91a-tls-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:40:08.282529 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.282304    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb2tn\" (UniqueName: \"kubernetes.io/projected/4366fe17-ae3c-4824-a12a-5cceed82a91a-kube-api-access-zb2tn\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:40:08.315820 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.315784    2554 generic.go:356] "Generic (PLEG): container finished" podID="4366fe17-ae3c-4824-a12a-5cceed82a91a" containerID="102573aa8f556c50991f579f27450e4397720cde39a2b98a6dea99b1a51bbbe7" exitCode=0
Sep 16 18:40:08.315979 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.315837    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"
Sep 16 18:40:08.315979 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.315870    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt" event={"ID":"4366fe17-ae3c-4824-a12a-5cceed82a91a","Type":"ContainerDied","Data":"102573aa8f556c50991f579f27450e4397720cde39a2b98a6dea99b1a51bbbe7"}
Sep 16 18:40:08.315979 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.315915    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt" event={"ID":"4366fe17-ae3c-4824-a12a-5cceed82a91a","Type":"ContainerDied","Data":"a373fb31d7f8cb28e9a965cb5ef8b32808c5571fe435dc82362432aafbba3100"}
Sep 16 18:40:08.315979 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.315937    2554 scope.go:117] "RemoveContainer" containerID="102573aa8f556c50991f579f27450e4397720cde39a2b98a6dea99b1a51bbbe7"
Sep 16 18:40:08.325391 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.325372    2554 scope.go:117] "RemoveContainer" containerID="102573aa8f556c50991f579f27450e4397720cde39a2b98a6dea99b1a51bbbe7"
Sep 16 18:40:08.325681 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:40:08.325662    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"102573aa8f556c50991f579f27450e4397720cde39a2b98a6dea99b1a51bbbe7\": container with ID starting with 102573aa8f556c50991f579f27450e4397720cde39a2b98a6dea99b1a51bbbe7 not found: ID does not exist" containerID="102573aa8f556c50991f579f27450e4397720cde39a2b98a6dea99b1a51bbbe7"
Sep 16 18:40:08.325735 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.325686    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102573aa8f556c50991f579f27450e4397720cde39a2b98a6dea99b1a51bbbe7"} err="failed to get container status \"102573aa8f556c50991f579f27450e4397720cde39a2b98a6dea99b1a51bbbe7\": rpc error: code = NotFound desc = could not find container \"102573aa8f556c50991f579f27450e4397720cde39a2b98a6dea99b1a51bbbe7\": container with ID starting with 102573aa8f556c50991f579f27450e4397720cde39a2b98a6dea99b1a51bbbe7 not found: ID does not exist"
Sep 16 18:40:08.339261 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.339219    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"]
Sep 16 18:40:08.345553 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.345528    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-a462d953/e2e-lifecycle-model-a462d953-kserve-66777c98b7-gqlnt"]
Sep 16 18:40:08.630331 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.630295    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"]
Sep 16 18:40:08.630796 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.630780    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0" containerName="payload-processing"
Sep 16 18:40:08.630897 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.630800    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0" containerName="payload-processing"
Sep 16 18:40:08.630897 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.630820    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5fcc9a-d49b-4c58-afc6-f1c74be355d1" containerName="payload-pre-processing"
Sep 16 18:40:08.630897 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.630828    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5fcc9a-d49b-4c58-afc6-f1c74be355d1" containerName="payload-pre-processing"
Sep 16 18:40:08.630897 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.630866    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4366fe17-ae3c-4824-a12a-5cceed82a91a" containerName="main"
Sep 16 18:40:08.630897 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.630875    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4366fe17-ae3c-4824-a12a-5cceed82a91a" containerName="main"
Sep 16 18:40:08.631144 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.630974    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e0dfcd1-df6d-46ce-b36a-78fbb54e48e0" containerName="payload-processing"
Sep 16 18:40:08.631144 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.630989    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab5fcc9a-d49b-4c58-afc6-f1c74be355d1" containerName="payload-pre-processing"
Sep 16 18:40:08.631144 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.630998    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4366fe17-ae3c-4824-a12a-5cceed82a91a" containerName="main"
Sep 16 18:40:08.635512 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.635491    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.638154 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.638131    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-972df0\"/\"openshift-service-ca.crt\""
Sep 16 18:40:08.639001 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.638985    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-972df0\"/\"auth-test-model-972df0-kserve-self-signed-certs\""
Sep 16 18:40:08.639108 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.639007    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-972df0\"/\"kube-root-ca.crt\""
Sep 16 18:40:08.639108 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.639011    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-972df0\"/\"default-dockercfg-hpjnn\""
Sep 16 18:40:08.646178 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.646154    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"]
Sep 16 18:40:08.686597 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.686561    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-dshm\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.686827 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.686615    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fhcv\" (UniqueName: \"kubernetes.io/projected/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-kube-api-access-5fhcv\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.686827 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.686711    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-home\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.686972 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.686835    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-tls-certs\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.686972 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.686866    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-tmp-dir\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.686972 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.686962    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-model-cache\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.788577 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.788541    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-tls-certs\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.788757 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.788586    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-tmp-dir\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.788757 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.788642    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-model-cache\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.788757 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.788669    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-dshm\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.788757 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.788689    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fhcv\" (UniqueName: \"kubernetes.io/projected/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-kube-api-access-5fhcv\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.788757 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.788735    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-home\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.789101 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.789074    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-model-cache\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.789162 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.789148    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-tmp-dir\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.789202 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.789160    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-home\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.790992 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.790968    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-dshm\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.791280 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.791263    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-tls-certs\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.798141 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.798115    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fhcv\" (UniqueName: \"kubernetes.io/projected/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-kube-api-access-5fhcv\") pod \"auth-test-model-972df0-kserve-69b5858959-hdlmr\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") " pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:08.885225 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.885134    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4366fe17-ae3c-4824-a12a-5cceed82a91a" path="/var/lib/kubelet/pods/4366fe17-ae3c-4824-a12a-5cceed82a91a/volumes"
Sep 16 18:40:08.946961 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:08.946912    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:09.227900 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:09.227874    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"]
Sep 16 18:40:09.229999 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:40:09.229972    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0326dcc8_4daf_4971_bea8_7fb756f5b7fa.slice/crio-90231f83b197d45a094ee4f33e178977b8296987c4a42923f397b42423ec36ba WatchSource:0}: Error finding container 90231f83b197d45a094ee4f33e178977b8296987c4a42923f397b42423ec36ba: Status 404 returned error can't find the container with id 90231f83b197d45a094ee4f33e178977b8296987c4a42923f397b42423ec36ba
Sep 16 18:40:09.330140 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:09.330091    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr" event={"ID":"0326dcc8-4daf-4971-bea8-7fb756f5b7fa","Type":"ContainerStarted","Data":"0c6a0e00c117947bcfbaf6e04ab1526866803e038df723962eb6b2edeb745a22"}
Sep 16 18:40:09.330140 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:09.330141    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr" event={"ID":"0326dcc8-4daf-4971-bea8-7fb756f5b7fa","Type":"ContainerStarted","Data":"90231f83b197d45a094ee4f33e178977b8296987c4a42923f397b42423ec36ba"}
Sep 16 18:40:09.352704 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:09.352641    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr" podStartSLOduration=1.352623489 podStartE2EDuration="1.352623489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:09.35032456 +0000 UTC m=+1432.909492385" watchObservedRunningTime="2026-09-16 18:40:09.352623489 +0000 UTC m=+1432.911791380"
Sep 16 18:40:18.948122 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:18.948080    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:18.948601 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:18.948143    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:18.960644 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:18.960612    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:19.398489 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:19.398455    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:40:23.919021 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:23.918982    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:40:53.731929 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:40:53.731889    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:41:16.896176 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:16.896149    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:41:16.901089 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:16.901059    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:41:30.063203 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:30.063172    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:41:32.293487 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.293449    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"]
Sep 16 18:41:32.297715 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.297691    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.314323 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.314299    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6942a4\"/\"kube-root-ca.crt\""
Sep 16 18:41:32.314471 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.314364    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6942a4\"/\"default-dockercfg-bmh8t\""
Sep 16 18:41:32.315135 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.315119    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6942a4\"/\"distinct-2-w4-6942a4-kserve-self-signed-certs\""
Sep 16 18:41:32.315186 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.315135    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6942a4\"/\"openshift-service-ca.crt\""
Sep 16 18:41:32.433273 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.433183    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-model-cache\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.433273 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.433257    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-tmp-dir\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.433521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.433356    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-home\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.433521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.433401    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-dshm\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.433521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.433446    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66b286d1-8e74-4b75-87ee-5d8213480721-tls-certs\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.433644 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.433534    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zpj7\" (UniqueName: \"kubernetes.io/projected/66b286d1-8e74-4b75-87ee-5d8213480721-kube-api-access-9zpj7\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.435391 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.435370    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"]
Sep 16 18:41:32.534094 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.534057    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-dshm\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.534299 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.534103    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66b286d1-8e74-4b75-87ee-5d8213480721-tls-certs\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.534299 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.534157    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zpj7\" (UniqueName: \"kubernetes.io/projected/66b286d1-8e74-4b75-87ee-5d8213480721-kube-api-access-9zpj7\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.534299 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.534211    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-model-cache\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.534421 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.534362    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-tmp-dir\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.534468 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.534420    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-home\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.534689 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.534669    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-model-cache\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.534781 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.534720    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-tmp-dir\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.534840 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.534777    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-home\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.536434 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.536413    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-dshm\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.536677 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.536660    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66b286d1-8e74-4b75-87ee-5d8213480721-tls-certs\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.557226 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.557155    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zpj7\" (UniqueName: \"kubernetes.io/projected/66b286d1-8e74-4b75-87ee-5d8213480721-kube-api-access-9zpj7\") pod \"distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") " pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.607222 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.607186    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:32.932193 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:32.931987    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"]
Sep 16 18:41:32.934967 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:41:32.934936    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66b286d1_8e74_4b75_87ee_5d8213480721.slice/crio-54ac1f0654676b01aed3419011123b276785a713bb06937bfa37409f40cfe87c WatchSource:0}: Error finding container 54ac1f0654676b01aed3419011123b276785a713bb06937bfa37409f40cfe87c: Status 404 returned error can't find the container with id 54ac1f0654676b01aed3419011123b276785a713bb06937bfa37409f40cfe87c
Sep 16 18:41:33.802300 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:33.802267    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d" event={"ID":"66b286d1-8e74-4b75-87ee-5d8213480721","Type":"ContainerStarted","Data":"bdb5a2c9f5d3a0020ac8c46cb4d5dcec484889f9293816f4aa720bbdb4691017"}
Sep 16 18:41:33.802659 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:33.802310    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d" event={"ID":"66b286d1-8e74-4b75-87ee-5d8213480721","Type":"ContainerStarted","Data":"54ac1f0654676b01aed3419011123b276785a713bb06937bfa37409f40cfe87c"}
Sep 16 18:41:33.843960 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:33.843904    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d" podStartSLOduration=1.8438874859999999 podStartE2EDuration="1.843887486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:33.838637413 +0000 UTC m=+1517.397805247" watchObservedRunningTime="2026-09-16 18:41:33.843887486 +0000 UTC m=+1517.403055365"
Sep 16 18:41:42.608276 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:42.608170    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:42.608276 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:42.608226    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:42.620799 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:42.620770    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:41:42.865481 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:41:42.865385    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:42:03.153712 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:03.153675    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:42:22.732550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.732516    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"]
Sep 16 18:42:22.736011 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.735989    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.739662 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.739642    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6a7741-openshift-default-dockercfg-vgtpw\""
Sep 16 18:42:22.764789 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.764759    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"]
Sep 16 18:42:22.830023 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.829985    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-istio-data\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.830023 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.830021    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/400fa802-1bf9-4020-90a6-639087dfafa7-istio-token\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.830278 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.830039    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq26t\" (UniqueName: \"kubernetes.io/projected/400fa802-1bf9-4020-90a6-639087dfafa7-kube-api-access-pq26t\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.830278 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.830135    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-workload-certs\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.830278 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.830167    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-credential-socket\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.830278 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.830198    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/400fa802-1bf9-4020-90a6-639087dfafa7-istiod-ca-cert\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.830500 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.830303    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-istio-envoy\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.830500 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.830352    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-workload-socket\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.830500 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.830412    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/400fa802-1bf9-4020-90a6-639087dfafa7-istio-podinfo\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.930956 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.930917    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-credential-socket\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.931143 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.930970    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/400fa802-1bf9-4020-90a6-639087dfafa7-istiod-ca-cert\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.931143 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.930996    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-istio-envoy\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.931143 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.931119    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-workload-socket\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.931354 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.931193    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/400fa802-1bf9-4020-90a6-639087dfafa7-istio-podinfo\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.931354 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.931283    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-istio-data\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.931354 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.931317    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/400fa802-1bf9-4020-90a6-639087dfafa7-istio-token\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.931354 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.931342    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq26t\" (UniqueName: \"kubernetes.io/projected/400fa802-1bf9-4020-90a6-639087dfafa7-kube-api-access-pq26t\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.931550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.931349    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-credential-socket\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.931550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.931443    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-workload-certs\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.931550 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.931527    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-workload-socket\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.931724 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.931704    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/400fa802-1bf9-4020-90a6-639087dfafa7-istiod-ca-cert\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.931870 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.931850    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-istio-data\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.931982 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.931967    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-workload-certs\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.933363 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.933344    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-istio-envoy\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.933867 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.933839    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/400fa802-1bf9-4020-90a6-639087dfafa7-istio-podinfo\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.942940 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.942918    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq26t\" (UniqueName: \"kubernetes.io/projected/400fa802-1bf9-4020-90a6-639087dfafa7-kube-api-access-pq26t\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:22.943893 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:22.943871    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/400fa802-1bf9-4020-90a6-639087dfafa7-istio-token\") pod \"e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") " pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:23.048303 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:23.048197    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:23.329641 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:23.329568    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"]
Sep 16 18:42:23.332816 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:42:23.332790    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400fa802_1bf9_4020_90a6_639087dfafa7.slice/crio-b8569ef9ea7941888005f7e224827d3f2ff2659722c6aea996ec43aeb138d9bb WatchSource:0}: Error finding container b8569ef9ea7941888005f7e224827d3f2ff2659722c6aea996ec43aeb138d9bb: Status 404 returned error can't find the container with id b8569ef9ea7941888005f7e224827d3f2ff2659722c6aea996ec43aeb138d9bb
Sep 16 18:42:23.335045 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:23.335013    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:42:23.335125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:23.335076    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:42:23.335125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:23.335106    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:42:24.092947 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:24.092908    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6" event={"ID":"400fa802-1bf9-4020-90a6-639087dfafa7","Type":"ContainerStarted","Data":"03e3a6fe890ef504cc7cbc4f285680452e4e758d157e0dba5b81c90270a0d2a8"}
Sep 16 18:42:24.093347 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:24.092974    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6" event={"ID":"400fa802-1bf9-4020-90a6-639087dfafa7","Type":"ContainerStarted","Data":"b8569ef9ea7941888005f7e224827d3f2ff2659722c6aea996ec43aeb138d9bb"}
Sep 16 18:42:24.115391 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:24.115339    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6" podStartSLOduration=2.115324267 podStartE2EDuration="2.115324267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:24.113587507 +0000 UTC m=+1567.672755377" watchObservedRunningTime="2026-09-16 18:42:24.115324267 +0000 UTC m=+1567.674492100"
Sep 16 18:42:25.048417 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:25.048382    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:25.054396 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:25.054369    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:25.098149 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:25.098121    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:25.099080 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:25.099059    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:42:32.329250 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:32.329207    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"]
Sep 16 18:42:32.333455 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:32.333430    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"
Sep 16 18:42:32.336310 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:32.336283    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6a7741\""
Sep 16 18:42:32.336555 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:32.336535    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6a7741-dockercfg-7p9gr\""
Sep 16 18:42:32.346443 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:32.346417    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"]
Sep 16 18:42:32.435894 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:32.435857    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a4890a-36b1-4233-9e3d-e50cd10a2f21-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq\" (UID: \"70a4890a-36b1-4233-9e3d-e50cd10a2f21\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"
Sep 16 18:42:32.436057 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:32.436004    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n9cp\" (UniqueName: \"kubernetes.io/projected/70a4890a-36b1-4233-9e3d-e50cd10a2f21-kube-api-access-5n9cp\") pod \"payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq\" (UID: \"70a4890a-36b1-4233-9e3d-e50cd10a2f21\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"
Sep 16 18:42:32.537334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:32.537296    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n9cp\" (UniqueName: \"kubernetes.io/projected/70a4890a-36b1-4233-9e3d-e50cd10a2f21-kube-api-access-5n9cp\") pod \"payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq\" (UID: \"70a4890a-36b1-4233-9e3d-e50cd10a2f21\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"
Sep 16 18:42:32.537504 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:32.537376    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a4890a-36b1-4233-9e3d-e50cd10a2f21-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq\" (UID: \"70a4890a-36b1-4233-9e3d-e50cd10a2f21\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"
Sep 16 18:42:32.538063 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:32.538042    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a4890a-36b1-4233-9e3d-e50cd10a2f21-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq\" (UID: \"70a4890a-36b1-4233-9e3d-e50cd10a2f21\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"
Sep 16 18:42:32.546129 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:32.546099    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n9cp\" (UniqueName: \"kubernetes.io/projected/70a4890a-36b1-4233-9e3d-e50cd10a2f21-kube-api-access-5n9cp\") pod \"payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq\" (UID: \"70a4890a-36b1-4233-9e3d-e50cd10a2f21\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"
Sep 16 18:42:32.647352 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:32.647264    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"
Sep 16 18:42:32.926972 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:32.926947    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"]
Sep 16 18:42:32.929161 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:42:32.929135    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a4890a_36b1_4233_9e3d_e50cd10a2f21.slice/crio-2e4afc8d758a53f01617b95604c347b358a4a9a1f3ae207716801b3ba21ef783 WatchSource:0}: Error finding container 2e4afc8d758a53f01617b95604c347b358a4a9a1f3ae207716801b3ba21ef783: Status 404 returned error can't find the container with id 2e4afc8d758a53f01617b95604c347b358a4a9a1f3ae207716801b3ba21ef783
Sep 16 18:42:33.145137 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:33.145092    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq" event={"ID":"70a4890a-36b1-4233-9e3d-e50cd10a2f21","Type":"ContainerStarted","Data":"de6367685853e48c5964932e562fba51ec2a268801684e636beb6eb8656291ca"}
Sep 16 18:42:33.145354 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:33.145178    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"
Sep 16 18:42:33.145354 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:33.145208    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq" event={"ID":"70a4890a-36b1-4233-9e3d-e50cd10a2f21","Type":"ContainerStarted","Data":"2e4afc8d758a53f01617b95604c347b358a4a9a1f3ae207716801b3ba21ef783"}
Sep 16 18:42:33.166977 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:33.166929    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq" podStartSLOduration=1.166914647 podStartE2EDuration="1.166914647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:33.164568253 +0000 UTC m=+1576.723736077" watchObservedRunningTime="2026-09-16 18:42:33.166914647 +0000 UTC m=+1576.726082519"
Sep 16 18:42:42.962671 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:42.962633    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:42:44.154199 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:44.154103    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"
Sep 16 18:42:45.214372 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:45.214342    2554 generic.go:356] "Generic (PLEG): container finished" podID="f5313a4b-6151-4a66-9688-b4c6b29f3a95" containerID="18ac9806dc04f192cf34ccfeced8bae480a03a74837d54c22d2f4249ac0aeb1b" exitCode=137
Sep 16 18:42:45.214768 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:45.214418    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln" event={"ID":"f5313a4b-6151-4a66-9688-b4c6b29f3a95","Type":"ContainerDied","Data":"18ac9806dc04f192cf34ccfeced8bae480a03a74837d54c22d2f4249ac0aeb1b"}
Sep 16 18:42:45.214768 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:45.214737    2554 scope.go:117] "RemoveContainer" containerID="18ac9806dc04f192cf34ccfeced8bae480a03a74837d54c22d2f4249ac0aeb1b"
Sep 16 18:42:46.222183 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:46.222147    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln" event={"ID":"f5313a4b-6151-4a66-9688-b4c6b29f3a95","Type":"ContainerStarted","Data":"93f36ec90b5acfb315db60d93b76ec1e101b6e9ae964f407a45984cb5e3e75df"}
Sep 16 18:42:46.222653 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:46.222403    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"
Sep 16 18:42:57.229381 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:42:57.229348    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"
Sep 16 18:43:03.853604 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:03.853565    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"]
Sep 16 18:43:03.858319 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:03.858292    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:43:03.863291 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:03.863267    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-61fa84-serving-cert\""
Sep 16 18:43:03.864983 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:03.864839    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"]
Sep 16 18:43:03.959366 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:03.959330    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6ebd759-e409-47b9-a680-60c080b6d96f-metrics-certs\") pod \"maas-api-e2e-api-b-61fa84-56b5f896f6-5svll\" (UID: \"c6ebd759-e409-47b9-a680-60c080b6d96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:43:03.959549 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:03.959456    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ebd759-e409-47b9-a680-60c080b6d96f-openshift-service-ca\") pod \"maas-api-e2e-api-b-61fa84-56b5f896f6-5svll\" (UID: \"c6ebd759-e409-47b9-a680-60c080b6d96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:43:03.959658 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:03.959638    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ebd759-e409-47b9-a680-60c080b6d96f-maas-api-tls\") pod \"maas-api-e2e-api-b-61fa84-56b5f896f6-5svll\" (UID: \"c6ebd759-e409-47b9-a680-60c080b6d96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:43:03.959705 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:03.959677    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgp5g\" (UniqueName: \"kubernetes.io/projected/c6ebd759-e409-47b9-a680-60c080b6d96f-kube-api-access-wgp5g\") pod \"maas-api-e2e-api-b-61fa84-56b5f896f6-5svll\" (UID: \"c6ebd759-e409-47b9-a680-60c080b6d96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:43:04.061318 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.061283    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6ebd759-e409-47b9-a680-60c080b6d96f-metrics-certs\") pod \"maas-api-e2e-api-b-61fa84-56b5f896f6-5svll\" (UID: \"c6ebd759-e409-47b9-a680-60c080b6d96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:43:04.061504 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.061366    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ebd759-e409-47b9-a680-60c080b6d96f-openshift-service-ca\") pod \"maas-api-e2e-api-b-61fa84-56b5f896f6-5svll\" (UID: \"c6ebd759-e409-47b9-a680-60c080b6d96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:43:04.061504 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.061474    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ebd759-e409-47b9-a680-60c080b6d96f-maas-api-tls\") pod \"maas-api-e2e-api-b-61fa84-56b5f896f6-5svll\" (UID: \"c6ebd759-e409-47b9-a680-60c080b6d96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:43:04.061504 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.061499    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgp5g\" (UniqueName: \"kubernetes.io/projected/c6ebd759-e409-47b9-a680-60c080b6d96f-kube-api-access-wgp5g\") pod \"maas-api-e2e-api-b-61fa84-56b5f896f6-5svll\" (UID: \"c6ebd759-e409-47b9-a680-60c080b6d96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:43:04.062028 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.062004    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ebd759-e409-47b9-a680-60c080b6d96f-openshift-service-ca\") pod \"maas-api-e2e-api-b-61fa84-56b5f896f6-5svll\" (UID: \"c6ebd759-e409-47b9-a680-60c080b6d96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:43:04.063667 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.063634    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ebd759-e409-47b9-a680-60c080b6d96f-maas-api-tls\") pod \"maas-api-e2e-api-b-61fa84-56b5f896f6-5svll\" (UID: \"c6ebd759-e409-47b9-a680-60c080b6d96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:43:04.063762 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.063718    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6ebd759-e409-47b9-a680-60c080b6d96f-metrics-certs\") pod \"maas-api-e2e-api-b-61fa84-56b5f896f6-5svll\" (UID: \"c6ebd759-e409-47b9-a680-60c080b6d96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:43:04.069989 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.069970    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgp5g\" (UniqueName: \"kubernetes.io/projected/c6ebd759-e409-47b9-a680-60c080b6d96f-kube-api-access-wgp5g\") pod \"maas-api-e2e-api-b-61fa84-56b5f896f6-5svll\" (UID: \"c6ebd759-e409-47b9-a680-60c080b6d96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:43:04.172676 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.172589    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:43:04.448479 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.448444    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"]
Sep 16 18:43:04.450045 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:43:04.450017    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ebd759_e409_47b9_a680_60c080b6d96f.slice/crio-a8f88d414f92d4df65294846d50dfa9b0b6326e501d2bf402475ab024bc40b62 WatchSource:0}: Error finding container a8f88d414f92d4df65294846d50dfa9b0b6326e501d2bf402475ab024bc40b62: Status 404 returned error can't find the container with id a8f88d414f92d4df65294846d50dfa9b0b6326e501d2bf402475ab024bc40b62
Sep 16 18:43:04.451690 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.451667    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:43:04.515758 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.512198    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"]
Sep 16 18:43:04.519284 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.519253    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"
Sep 16 18:43:04.522069 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.522036    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-61fa84\""
Sep 16 18:43:04.522207 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.522099    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-61fa84-dockercfg-d868m\""
Sep 16 18:43:04.522953 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.522932    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"]
Sep 16 18:43:04.567331 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.567296    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v72d7\" (UniqueName: \"kubernetes.io/projected/5abcadd6-15bc-4bf1-8d39-7fa8162eba56-kube-api-access-v72d7\") pod \"payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r\" (UID: \"5abcadd6-15bc-4bf1-8d39-7fa8162eba56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"
Sep 16 18:43:04.567503 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.567449    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5abcadd6-15bc-4bf1-8d39-7fa8162eba56-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r\" (UID: \"5abcadd6-15bc-4bf1-8d39-7fa8162eba56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"
Sep 16 18:43:04.668320 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.668272    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5abcadd6-15bc-4bf1-8d39-7fa8162eba56-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r\" (UID: \"5abcadd6-15bc-4bf1-8d39-7fa8162eba56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"
Sep 16 18:43:04.668489 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.668391    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v72d7\" (UniqueName: \"kubernetes.io/projected/5abcadd6-15bc-4bf1-8d39-7fa8162eba56-kube-api-access-v72d7\") pod \"payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r\" (UID: \"5abcadd6-15bc-4bf1-8d39-7fa8162eba56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"
Sep 16 18:43:04.669206 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.669178    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5abcadd6-15bc-4bf1-8d39-7fa8162eba56-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r\" (UID: \"5abcadd6-15bc-4bf1-8d39-7fa8162eba56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"
Sep 16 18:43:04.677538 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.677513    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v72d7\" (UniqueName: \"kubernetes.io/projected/5abcadd6-15bc-4bf1-8d39-7fa8162eba56-kube-api-access-v72d7\") pod \"payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r\" (UID: \"5abcadd6-15bc-4bf1-8d39-7fa8162eba56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"
Sep 16 18:43:04.831150 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:04.831126    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"
Sep 16 18:43:05.110421 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:05.110396    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"]
Sep 16 18:43:05.112727 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:43:05.112694    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abcadd6_15bc_4bf1_8d39_7fa8162eba56.slice/crio-33f7cf6cb067f5aff3a9cd3bb6e89988f910856a9c1b283c7048f07dc82ea779 WatchSource:0}: Error finding container 33f7cf6cb067f5aff3a9cd3bb6e89988f910856a9c1b283c7048f07dc82ea779: Status 404 returned error can't find the container with id 33f7cf6cb067f5aff3a9cd3bb6e89988f910856a9c1b283c7048f07dc82ea779
Sep 16 18:43:05.326498 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:05.326463    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r" event={"ID":"5abcadd6-15bc-4bf1-8d39-7fa8162eba56","Type":"ContainerStarted","Data":"63e76f89712054ff6676825d7fd16665f615847c5229e81690193383fceb58a6"}
Sep 16 18:43:05.326680 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:05.326519    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r" event={"ID":"5abcadd6-15bc-4bf1-8d39-7fa8162eba56","Type":"ContainerStarted","Data":"33f7cf6cb067f5aff3a9cd3bb6e89988f910856a9c1b283c7048f07dc82ea779"}
Sep 16 18:43:05.326680 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:05.326577    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"
Sep 16 18:43:05.328130 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:05.328100    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll" event={"ID":"c6ebd759-e409-47b9-a680-60c080b6d96f","Type":"ContainerStarted","Data":"2b88dd3f19168e2039fb0c398c5e0f864f9ddbd7c03f59a99d04e0d9f7e84e08"}
Sep 16 18:43:05.328252 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:05.328151    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll" event={"ID":"c6ebd759-e409-47b9-a680-60c080b6d96f","Type":"ContainerStarted","Data":"a8f88d414f92d4df65294846d50dfa9b0b6326e501d2bf402475ab024bc40b62"}
Sep 16 18:43:05.328252 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:05.328195    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:43:05.345822 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:05.345768    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r" podStartSLOduration=1.3457546009999999 podStartE2EDuration="1.345754601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:05.345523089 +0000 UTC m=+1608.904690921" watchObservedRunningTime="2026-09-16 18:43:05.345754601 +0000 UTC m=+1608.904922434"
Sep 16 18:43:05.367693 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:05.367607    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll" podStartSLOduration=2.040804673 podStartE2EDuration="2.367587648s" totalImagesPullingTime="326.782975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:05.365115813 +0000 UTC m=+1608.924283646" watchObservedRunningTime="2026-09-16 18:43:05.367587648 +0000 UTC m=+1608.926755483"
Sep 16 18:43:11.339677 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:11.339596    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:43:16.336165 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:16.336128    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"
Sep 16 18:43:26.105604 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:26.105388    2554 patch_prober.go:28] interesting pod/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 18:43:26.105604 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:26.105489    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6" podUID="400fa802-1bf9-4020-90a6-639087dfafa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 18:43:26.451507 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:26.451413    2554 generic.go:356] "Generic (PLEG): container finished" podID="400fa802-1bf9-4020-90a6-639087dfafa7" containerID="03e3a6fe890ef504cc7cbc4f285680452e4e758d157e0dba5b81c90270a0d2a8" exitCode=0
Sep 16 18:43:26.451645 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:26.451492    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6" event={"ID":"400fa802-1bf9-4020-90a6-639087dfafa7","Type":"ContainerDied","Data":"03e3a6fe890ef504cc7cbc4f285680452e4e758d157e0dba5b81c90270a0d2a8"}
Sep 16 18:43:26.452061 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:26.452045    2554 scope.go:117] "RemoveContainer" containerID="03e3a6fe890ef504cc7cbc4f285680452e4e758d157e0dba5b81c90270a0d2a8"
Sep 16 18:43:26.453349 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:26.453317    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:43:26.453404 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:26.453387    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:43:26.453438 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:26.453415    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:43:27.466430 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:27.466384    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6" event={"ID":"400fa802-1bf9-4020-90a6-639087dfafa7","Type":"ContainerStarted","Data":"e94b5f6d8f395e3cc186f62d7b07e7723de7e941befcdbdf99661306bad0c01f"}
Sep 16 18:43:28.048687 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:28.048648    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:43:28.048903 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:28.048701    2554 patch_prober.go:28] interesting pod/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 16 18:43:28.048903 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:28.048763    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6" podUID="400fa802-1bf9-4020-90a6-639087dfafa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 16 18:43:28.543368 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:28.543330    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"]
Sep 16 18:43:29.049613 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:29.049576    2554 patch_prober.go:28] interesting pod/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 16 18:43:29.049802 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:29.049643    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6" podUID="400fa802-1bf9-4020-90a6-639087dfafa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 16 18:43:30.048940 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:30.048893    2554 patch_prober.go:28] interesting pod/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 16 18:43:30.049418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:30.048996    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6" podUID="400fa802-1bf9-4020-90a6-639087dfafa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 16 18:43:31.048921 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:31.048878    2554 patch_prober.go:28] interesting pod/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 16 18:43:31.049125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:31.048950    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6" podUID="400fa802-1bf9-4020-90a6-639087dfafa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 16 18:43:32.048924 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:32.048876    2554 patch_prober.go:28] interesting pod/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 16 18:43:32.049145 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:32.048948    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6" podUID="400fa802-1bf9-4020-90a6-639087dfafa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 16 18:43:33.052781 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:33.052746    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:43:33.053209 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:33.053134    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:43:33.053716 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:33.053696    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:43:34.308858 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:34.308819    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:43:43.581330 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.581292    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g" podUID="bee55eb5-c320-4715-b2a9-6c8b6ed4d800" containerName="main" containerID="cri-o://c31d6978ef8e0258450612fec2413e34f8b7b60a7fc909dbe70fc25f5cdca129" gracePeriod=45
Sep 16 18:43:43.841178 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.841101    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:43:43.895454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.895425    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-model-cache\") pod \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") "
Sep 16 18:43:43.895614 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.895494    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-home\") pod \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") "
Sep 16 18:43:43.895614 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.895556    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-tmp-dir\") pod \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") "
Sep 16 18:43:43.895614 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.895596    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-dshm\") pod \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") "
Sep 16 18:43:43.895755 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.895648    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-tls-certs\") pod \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") "
Sep 16 18:43:43.895755 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.895694    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvcwg\" (UniqueName: \"kubernetes.io/projected/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-kube-api-access-bvcwg\") pod \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\" (UID: \"bee55eb5-c320-4715-b2a9-6c8b6ed4d800\") "
Sep 16 18:43:43.895864 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.895782    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-model-cache" (OuterVolumeSpecName: "model-cache") pod "bee55eb5-c320-4715-b2a9-6c8b6ed4d800" (UID: "bee55eb5-c320-4715-b2a9-6c8b6ed4d800"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:43.895864 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.895782    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-home" (OuterVolumeSpecName: "home") pod "bee55eb5-c320-4715-b2a9-6c8b6ed4d800" (UID: "bee55eb5-c320-4715-b2a9-6c8b6ed4d800"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:43.895864 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.895791    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bee55eb5-c320-4715-b2a9-6c8b6ed4d800" (UID: "bee55eb5-c320-4715-b2a9-6c8b6ed4d800"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:43.896379 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.896060    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-model-cache\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:43.896379 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.896080    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-home\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:43.896379 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.896094    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-tmp-dir\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:43.897937 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.897906    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-dshm" (OuterVolumeSpecName: "dshm") pod "bee55eb5-c320-4715-b2a9-6c8b6ed4d800" (UID: "bee55eb5-c320-4715-b2a9-6c8b6ed4d800"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:43.898045 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.897987    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-kube-api-access-bvcwg" (OuterVolumeSpecName: "kube-api-access-bvcwg") pod "bee55eb5-c320-4715-b2a9-6c8b6ed4d800" (UID: "bee55eb5-c320-4715-b2a9-6c8b6ed4d800"). InnerVolumeSpecName "kube-api-access-bvcwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:43.898126 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.898107    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bee55eb5-c320-4715-b2a9-6c8b6ed4d800" (UID: "bee55eb5-c320-4715-b2a9-6c8b6ed4d800"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:43.997139 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.997108    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-dshm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:43.997139 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.997136    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-tls-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:43.997139 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:43.997148    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvcwg\" (UniqueName: \"kubernetes.io/projected/bee55eb5-c320-4715-b2a9-6c8b6ed4d800-kube-api-access-bvcwg\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:44.380649 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.380611    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"]
Sep 16 18:43:44.380959 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.380931    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz" podUID="32667ec6-64cb-4b8f-87a2-87e7924794ed" containerName="payload-processing" containerID="cri-o://938ae6aea147472e421fbde33946611826bc56a19c7ca3ce893c8e066208eb91" gracePeriod=30
Sep 16 18:43:44.570380 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.570345    2554 generic.go:356] "Generic (PLEG): container finished" podID="32667ec6-64cb-4b8f-87a2-87e7924794ed" containerID="938ae6aea147472e421fbde33946611826bc56a19c7ca3ce893c8e066208eb91" exitCode=0
Sep 16 18:43:44.570541 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.570420    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz" event={"ID":"32667ec6-64cb-4b8f-87a2-87e7924794ed","Type":"ContainerDied","Data":"938ae6aea147472e421fbde33946611826bc56a19c7ca3ce893c8e066208eb91"}
Sep 16 18:43:44.572558 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.572532    2554 generic.go:356] "Generic (PLEG): container finished" podID="bee55eb5-c320-4715-b2a9-6c8b6ed4d800" containerID="c31d6978ef8e0258450612fec2413e34f8b7b60a7fc909dbe70fc25f5cdca129" exitCode=0
Sep 16 18:43:44.572694 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.572606    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"
Sep 16 18:43:44.572694 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.572614    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g" event={"ID":"bee55eb5-c320-4715-b2a9-6c8b6ed4d800","Type":"ContainerDied","Data":"c31d6978ef8e0258450612fec2413e34f8b7b60a7fc909dbe70fc25f5cdca129"}
Sep 16 18:43:44.572694 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.572670    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g" event={"ID":"bee55eb5-c320-4715-b2a9-6c8b6ed4d800","Type":"ContainerDied","Data":"3a2a395ff2390a4f45a3a56bf30a07b146133e181231c8b4053eb1c5f7235d24"}
Sep 16 18:43:44.572694 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.572694    2554 scope.go:117] "RemoveContainer" containerID="c31d6978ef8e0258450612fec2413e34f8b7b60a7fc909dbe70fc25f5cdca129"
Sep 16 18:43:44.589520 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.589452    2554 scope.go:117] "RemoveContainer" containerID="c31d6978ef8e0258450612fec2413e34f8b7b60a7fc909dbe70fc25f5cdca129"
Sep 16 18:43:44.590706 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:43:44.590572    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31d6978ef8e0258450612fec2413e34f8b7b60a7fc909dbe70fc25f5cdca129\": container with ID starting with c31d6978ef8e0258450612fec2413e34f8b7b60a7fc909dbe70fc25f5cdca129 not found: ID does not exist" containerID="c31d6978ef8e0258450612fec2413e34f8b7b60a7fc909dbe70fc25f5cdca129"
Sep 16 18:43:44.590706 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.590635    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31d6978ef8e0258450612fec2413e34f8b7b60a7fc909dbe70fc25f5cdca129"} err="failed to get container status \"c31d6978ef8e0258450612fec2413e34f8b7b60a7fc909dbe70fc25f5cdca129\": rpc error: code = NotFound desc = could not find container \"c31d6978ef8e0258450612fec2413e34f8b7b60a7fc909dbe70fc25f5cdca129\": container with ID starting with c31d6978ef8e0258450612fec2413e34f8b7b60a7fc909dbe70fc25f5cdca129 not found: ID does not exist"
Sep 16 18:43:44.639014 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.638973    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"]
Sep 16 18:43:44.650320 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.646802    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a24d09/distinct-2-w0-a24d09-kserve-db588ddcb-rfs4g"]
Sep 16 18:43:44.665103 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.665081    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"
Sep 16 18:43:44.705046 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.705010    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppnsc\" (UniqueName: \"kubernetes.io/projected/32667ec6-64cb-4b8f-87a2-87e7924794ed-kube-api-access-ppnsc\") pod \"32667ec6-64cb-4b8f-87a2-87e7924794ed\" (UID: \"32667ec6-64cb-4b8f-87a2-87e7924794ed\") "
Sep 16 18:43:44.705195 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.705151    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32667ec6-64cb-4b8f-87a2-87e7924794ed-plugins-config-volume\") pod \"32667ec6-64cb-4b8f-87a2-87e7924794ed\" (UID: \"32667ec6-64cb-4b8f-87a2-87e7924794ed\") "
Sep 16 18:43:44.705626 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.705598    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32667ec6-64cb-4b8f-87a2-87e7924794ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "32667ec6-64cb-4b8f-87a2-87e7924794ed" (UID: "32667ec6-64cb-4b8f-87a2-87e7924794ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:44.707382 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.707355    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32667ec6-64cb-4b8f-87a2-87e7924794ed-kube-api-access-ppnsc" (OuterVolumeSpecName: "kube-api-access-ppnsc") pod "32667ec6-64cb-4b8f-87a2-87e7924794ed" (UID: "32667ec6-64cb-4b8f-87a2-87e7924794ed"). InnerVolumeSpecName "kube-api-access-ppnsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:44.806500 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.806464    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32667ec6-64cb-4b8f-87a2-87e7924794ed-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:44.806500 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.806492    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppnsc\" (UniqueName: \"kubernetes.io/projected/32667ec6-64cb-4b8f-87a2-87e7924794ed-kube-api-access-ppnsc\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:44.884971 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:44.884892    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bee55eb5-c320-4715-b2a9-6c8b6ed4d800" path="/var/lib/kubelet/pods/bee55eb5-c320-4715-b2a9-6c8b6ed4d800/volumes"
Sep 16 18:43:45.579949 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:45.579917    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"
Sep 16 18:43:45.580138 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:45.579959    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz" event={"ID":"32667ec6-64cb-4b8f-87a2-87e7924794ed","Type":"ContainerDied","Data":"81156108db343f2f42d1d49380ba49a887ccdb1c67394f76bab735cf55d12bb8"}
Sep 16 18:43:45.580138 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:45.580022    2554 scope.go:117] "RemoveContainer" containerID="938ae6aea147472e421fbde33946611826bc56a19c7ca3ce893c8e066208eb91"
Sep 16 18:43:45.598346 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:45.598321    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"]
Sep 16 18:43:45.602579 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:45.602557    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a24d09-6cc9f769c5-nshdz"]
Sep 16 18:43:46.884637 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:46.884604    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32667ec6-64cb-4b8f-87a2-87e7924794ed" path="/var/lib/kubelet/pods/32667ec6-64cb-4b8f-87a2-87e7924794ed/volumes"
Sep 16 18:43:50.840903 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:50.840522    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"]
Sep 16 18:43:50.840903 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:50.840772    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq" podUID="70a4890a-36b1-4233-9e3d-e50cd10a2f21" containerName="payload-pre-processing" containerID="cri-o://de6367685853e48c5964932e562fba51ec2a268801684e636beb6eb8656291ca" gracePeriod=30
Sep 16 18:43:51.107185 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:51.107133    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"
Sep 16 18:43:51.182177 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:51.182134    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n9cp\" (UniqueName: \"kubernetes.io/projected/70a4890a-36b1-4233-9e3d-e50cd10a2f21-kube-api-access-5n9cp\") pod \"70a4890a-36b1-4233-9e3d-e50cd10a2f21\" (UID: \"70a4890a-36b1-4233-9e3d-e50cd10a2f21\") "
Sep 16 18:43:51.182177 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:51.182174    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a4890a-36b1-4233-9e3d-e50cd10a2f21-plugins-config-volume\") pod \"70a4890a-36b1-4233-9e3d-e50cd10a2f21\" (UID: \"70a4890a-36b1-4233-9e3d-e50cd10a2f21\") "
Sep 16 18:43:51.182631 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:51.182605    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a4890a-36b1-4233-9e3d-e50cd10a2f21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70a4890a-36b1-4233-9e3d-e50cd10a2f21" (UID: "70a4890a-36b1-4233-9e3d-e50cd10a2f21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:51.184702 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:51.184661    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a4890a-36b1-4233-9e3d-e50cd10a2f21-kube-api-access-5n9cp" (OuterVolumeSpecName: "kube-api-access-5n9cp") pod "70a4890a-36b1-4233-9e3d-e50cd10a2f21" (UID: "70a4890a-36b1-4233-9e3d-e50cd10a2f21"). InnerVolumeSpecName "kube-api-access-5n9cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:51.283350 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:51.283312    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n9cp\" (UniqueName: \"kubernetes.io/projected/70a4890a-36b1-4233-9e3d-e50cd10a2f21-kube-api-access-5n9cp\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:51.283350 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:51.283344    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a4890a-36b1-4233-9e3d-e50cd10a2f21-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:51.617028 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:51.616988    2554 generic.go:356] "Generic (PLEG): container finished" podID="70a4890a-36b1-4233-9e3d-e50cd10a2f21" containerID="de6367685853e48c5964932e562fba51ec2a268801684e636beb6eb8656291ca" exitCode=0
Sep 16 18:43:51.617206 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:51.617034    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"
Sep 16 18:43:51.617206 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:51.617039    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq" event={"ID":"70a4890a-36b1-4233-9e3d-e50cd10a2f21","Type":"ContainerDied","Data":"de6367685853e48c5964932e562fba51ec2a268801684e636beb6eb8656291ca"}
Sep 16 18:43:51.617206 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:51.617158    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq" event={"ID":"70a4890a-36b1-4233-9e3d-e50cd10a2f21","Type":"ContainerDied","Data":"2e4afc8d758a53f01617b95604c347b358a4a9a1f3ae207716801b3ba21ef783"}
Sep 16 18:43:51.617206 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:51.617186    2554 scope.go:117] "RemoveContainer" containerID="de6367685853e48c5964932e562fba51ec2a268801684e636beb6eb8656291ca"
Sep 16 18:43:51.627253 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:51.627218    2554 scope.go:117] "RemoveContainer" containerID="de6367685853e48c5964932e562fba51ec2a268801684e636beb6eb8656291ca"
Sep 16 18:43:51.627538 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:43:51.627515    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6367685853e48c5964932e562fba51ec2a268801684e636beb6eb8656291ca\": container with ID starting with de6367685853e48c5964932e562fba51ec2a268801684e636beb6eb8656291ca not found: ID does not exist" containerID="de6367685853e48c5964932e562fba51ec2a268801684e636beb6eb8656291ca"
Sep 16 18:43:51.627612 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:51.627548    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6367685853e48c5964932e562fba51ec2a268801684e636beb6eb8656291ca"} err="failed to get container status \"de6367685853e48c5964932e562fba51ec2a268801684e636beb6eb8656291ca\": rpc error: code = NotFound desc = could not find container \"de6367685853e48c5964932e562fba51ec2a268801684e636beb6eb8656291ca\": container with ID starting with de6367685853e48c5964932e562fba51ec2a268801684e636beb6eb8656291ca not found: ID does not exist"
Sep 16 18:43:51.643311 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:51.643287    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"]
Sep 16 18:43:51.650502 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:51.650481    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a7741-6f6d68694-9cfjq"]
Sep 16 18:43:52.883891 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:52.883861    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70a4890a-36b1-4233-9e3d-e50cd10a2f21" path="/var/lib/kubelet/pods/70a4890a-36b1-4233-9e3d-e50cd10a2f21/volumes"
Sep 16 18:43:53.530789 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:53.530754    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"]
Sep 16 18:43:53.531100 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:53.531067    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm" podUID="335a6bc4-c505-4fab-b12f-d2ccc04e971d" containerName="istio-proxy" containerID="cri-o://7de6264a99c2038d6575467655c9edbd7b05198e1f70c231a708edeab9b3244e" gracePeriod=30
Sep 16 18:43:58.100428 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.100395    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"]
Sep 16 18:43:58.100820 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.100609    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6" podUID="400fa802-1bf9-4020-90a6-639087dfafa7" containerName="istio-proxy" containerID="cri-o://e94b5f6d8f395e3cc186f62d7b07e7723de7e941befcdbdf99661306bad0c01f" gracePeriod=30
Sep 16 18:43:58.663413 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.663380    2554 generic.go:356] "Generic (PLEG): container finished" podID="335a6bc4-c505-4fab-b12f-d2ccc04e971d" containerID="7de6264a99c2038d6575467655c9edbd7b05198e1f70c231a708edeab9b3244e" exitCode=0
Sep 16 18:43:58.663570 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.663444    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm" event={"ID":"335a6bc4-c505-4fab-b12f-d2ccc04e971d","Type":"ContainerDied","Data":"7de6264a99c2038d6575467655c9edbd7b05198e1f70c231a708edeab9b3244e"}
Sep 16 18:43:58.663570 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.663523    2554 scope.go:117] "RemoveContainer" containerID="6f267a416e3af69e03dbdc905a8b756371124c27fe044f0c01a68b5410bb2668"
Sep 16 18:43:58.798515 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.798447    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:43:58.963521 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.963488    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istiod-ca-cert\") pod \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") "
Sep 16 18:43:58.963679 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.963556    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-envoy\") pod \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") "
Sep 16 18:43:58.963679 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.963595    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-token\") pod \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") "
Sep 16 18:43:58.963679 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.963616    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-credential-socket\") pod \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") "
Sep 16 18:43:58.963679 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.963676    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-podinfo\") pod \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") "
Sep 16 18:43:58.963864 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.963696    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-workload-certs\") pod \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") "
Sep 16 18:43:58.963864 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.963746    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-workload-socket\") pod \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") "
Sep 16 18:43:58.963864 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.963775    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-data\") pod \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") "
Sep 16 18:43:58.963864 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.963793    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dmkv\" (UniqueName: \"kubernetes.io/projected/335a6bc4-c505-4fab-b12f-d2ccc04e971d-kube-api-access-9dmkv\") pod \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\" (UID: \"335a6bc4-c505-4fab-b12f-d2ccc04e971d\") "
Sep 16 18:43:58.964060 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.963916    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "335a6bc4-c505-4fab-b12f-d2ccc04e971d" (UID: "335a6bc4-c505-4fab-b12f-d2ccc04e971d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:58.964060 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.964041    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "335a6bc4-c505-4fab-b12f-d2ccc04e971d" (UID: "335a6bc4-c505-4fab-b12f-d2ccc04e971d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:58.964183 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.964050    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "335a6bc4-c505-4fab-b12f-d2ccc04e971d" (UID: "335a6bc4-c505-4fab-b12f-d2ccc04e971d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:58.964268 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.964224    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-data" (OuterVolumeSpecName: "istio-data") pod "335a6bc4-c505-4fab-b12f-d2ccc04e971d" (UID: "335a6bc4-c505-4fab-b12f-d2ccc04e971d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:58.964375 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.964358    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:58.964435 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.964380    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:58.964435 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.964394    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:58.964435 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.964400    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "335a6bc4-c505-4fab-b12f-d2ccc04e971d" (UID: "335a6bc4-c505-4fab-b12f-d2ccc04e971d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:58.964435 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.964406    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:58.966015 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.965984    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-token" (OuterVolumeSpecName: "istio-token") pod "335a6bc4-c505-4fab-b12f-d2ccc04e971d" (UID: "335a6bc4-c505-4fab-b12f-d2ccc04e971d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:58.966015 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.966002    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "335a6bc4-c505-4fab-b12f-d2ccc04e971d" (UID: "335a6bc4-c505-4fab-b12f-d2ccc04e971d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:43:58.966161 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.966019    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335a6bc4-c505-4fab-b12f-d2ccc04e971d-kube-api-access-9dmkv" (OuterVolumeSpecName: "kube-api-access-9dmkv") pod "335a6bc4-c505-4fab-b12f-d2ccc04e971d" (UID: "335a6bc4-c505-4fab-b12f-d2ccc04e971d"). InnerVolumeSpecName "kube-api-access-9dmkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:58.966161 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:58.966104    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "335a6bc4-c505-4fab-b12f-d2ccc04e971d" (UID: "335a6bc4-c505-4fab-b12f-d2ccc04e971d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:59.065283 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:59.065179    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dmkv\" (UniqueName: \"kubernetes.io/projected/335a6bc4-c505-4fab-b12f-d2ccc04e971d-kube-api-access-9dmkv\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:59.065283 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:59.065212    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:59.065283 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:59.065221    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:59.065283 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:59.065260    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335a6bc4-c505-4fab-b12f-d2ccc04e971d-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:59.065283 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:59.065272    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335a6bc4-c505-4fab-b12f-d2ccc04e971d-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:43:59.670140 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:59.670106    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"
Sep 16 18:43:59.670640 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:59.670102    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm" event={"ID":"335a6bc4-c505-4fab-b12f-d2ccc04e971d","Type":"ContainerDied","Data":"c91a610431414ef2d4962192de2f66fec5c9218f5f3aee0e49bb5cfefe892f02"}
Sep 16 18:43:59.670640 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:59.670265    2554 scope.go:117] "RemoveContainer" containerID="7de6264a99c2038d6575467655c9edbd7b05198e1f70c231a708edeab9b3244e"
Sep 16 18:43:59.692444 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:59.692419    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"]
Sep 16 18:43:59.697328 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:43:59.697305    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a24d09-openshift-default-6bdd67b488-hztnm"]
Sep 16 18:44:00.884114 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:00.884081    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="335a6bc4-c505-4fab-b12f-d2ccc04e971d" path="/var/lib/kubelet/pods/335a6bc4-c505-4fab-b12f-d2ccc04e971d/volumes"
Sep 16 18:44:03.056197 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.056163    2554 patch_prober.go:28] interesting pod/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 18:44:03.056644 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.056224    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6" podUID="400fa802-1bf9-4020-90a6-639087dfafa7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:44:03.346383 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.346358    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:44:03.509973 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.509942    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/400fa802-1bf9-4020-90a6-639087dfafa7-istio-podinfo\") pod \"400fa802-1bf9-4020-90a6-639087dfafa7\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") "
Sep 16 18:44:03.509973 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.509984    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pq26t\" (UniqueName: \"kubernetes.io/projected/400fa802-1bf9-4020-90a6-639087dfafa7-kube-api-access-pq26t\") pod \"400fa802-1bf9-4020-90a6-639087dfafa7\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") "
Sep 16 18:44:03.510211 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.510012    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/400fa802-1bf9-4020-90a6-639087dfafa7-istiod-ca-cert\") pod \"400fa802-1bf9-4020-90a6-639087dfafa7\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") "
Sep 16 18:44:03.510211 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.510043    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-istio-data\") pod \"400fa802-1bf9-4020-90a6-639087dfafa7\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") "
Sep 16 18:44:03.510211 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.510106    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-workload-socket\") pod \"400fa802-1bf9-4020-90a6-639087dfafa7\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") "
Sep 16 18:44:03.510211 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.510150    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-credential-socket\") pod \"400fa802-1bf9-4020-90a6-639087dfafa7\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") "
Sep 16 18:44:03.510211 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.510172    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-workload-certs\") pod \"400fa802-1bf9-4020-90a6-639087dfafa7\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") "
Sep 16 18:44:03.510493 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.510214    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/400fa802-1bf9-4020-90a6-639087dfafa7-istio-token\") pod \"400fa802-1bf9-4020-90a6-639087dfafa7\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") "
Sep 16 18:44:03.510493 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.510279    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-istio-envoy\") pod \"400fa802-1bf9-4020-90a6-639087dfafa7\" (UID: \"400fa802-1bf9-4020-90a6-639087dfafa7\") "
Sep 16 18:44:03.510493 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.510373    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-istio-data" (OuterVolumeSpecName: "istio-data") pod "400fa802-1bf9-4020-90a6-639087dfafa7" (UID: "400fa802-1bf9-4020-90a6-639087dfafa7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:03.510493 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.510437    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "400fa802-1bf9-4020-90a6-639087dfafa7" (UID: "400fa802-1bf9-4020-90a6-639087dfafa7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:03.510691 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.510535    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400fa802-1bf9-4020-90a6-639087dfafa7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "400fa802-1bf9-4020-90a6-639087dfafa7" (UID: "400fa802-1bf9-4020-90a6-639087dfafa7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:03.510691 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.510606    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "400fa802-1bf9-4020-90a6-639087dfafa7" (UID: "400fa802-1bf9-4020-90a6-639087dfafa7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:03.510691 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.510660    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "400fa802-1bf9-4020-90a6-639087dfafa7" (UID: "400fa802-1bf9-4020-90a6-639087dfafa7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:03.510825 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.510714    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:03.510825 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.510734    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:03.512277 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.512210    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/400fa802-1bf9-4020-90a6-639087dfafa7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "400fa802-1bf9-4020-90a6-639087dfafa7" (UID: "400fa802-1bf9-4020-90a6-639087dfafa7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:44:03.512444 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.512421    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400fa802-1bf9-4020-90a6-639087dfafa7-kube-api-access-pq26t" (OuterVolumeSpecName: "kube-api-access-pq26t") pod "400fa802-1bf9-4020-90a6-639087dfafa7" (UID: "400fa802-1bf9-4020-90a6-639087dfafa7"). InnerVolumeSpecName "kube-api-access-pq26t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:03.512738 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.512717    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "400fa802-1bf9-4020-90a6-639087dfafa7" (UID: "400fa802-1bf9-4020-90a6-639087dfafa7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:03.512836 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.512718    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400fa802-1bf9-4020-90a6-639087dfafa7-istio-token" (OuterVolumeSpecName: "istio-token") pod "400fa802-1bf9-4020-90a6-639087dfafa7" (UID: "400fa802-1bf9-4020-90a6-639087dfafa7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:03.612017 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.611927    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:03.612017 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.611961    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:03.612017 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.611971    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/400fa802-1bf9-4020-90a6-639087dfafa7-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:03.612017 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.611980    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/400fa802-1bf9-4020-90a6-639087dfafa7-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:03.612017 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.611989    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/400fa802-1bf9-4020-90a6-639087dfafa7-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:03.612017 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.611996    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq26t\" (UniqueName: \"kubernetes.io/projected/400fa802-1bf9-4020-90a6-639087dfafa7-kube-api-access-pq26t\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:03.612017 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.612005    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/400fa802-1bf9-4020-90a6-639087dfafa7-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:03.696218 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.696181    2554 generic.go:356] "Generic (PLEG): container finished" podID="400fa802-1bf9-4020-90a6-639087dfafa7" containerID="e94b5f6d8f395e3cc186f62d7b07e7723de7e941befcdbdf99661306bad0c01f" exitCode=0
Sep 16 18:44:03.696218 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.696212    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6" event={"ID":"400fa802-1bf9-4020-90a6-639087dfafa7","Type":"ContainerDied","Data":"e94b5f6d8f395e3cc186f62d7b07e7723de7e941befcdbdf99661306bad0c01f"}
Sep 16 18:44:03.696437 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.696224    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"
Sep 16 18:44:03.696437 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.696299    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6" event={"ID":"400fa802-1bf9-4020-90a6-639087dfafa7","Type":"ContainerDied","Data":"b8569ef9ea7941888005f7e224827d3f2ff2659722c6aea996ec43aeb138d9bb"}
Sep 16 18:44:03.696437 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.696324    2554 scope.go:117] "RemoveContainer" containerID="e94b5f6d8f395e3cc186f62d7b07e7723de7e941befcdbdf99661306bad0c01f"
Sep 16 18:44:03.706562 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.706544    2554 scope.go:117] "RemoveContainer" containerID="03e3a6fe890ef504cc7cbc4f285680452e4e758d157e0dba5b81c90270a0d2a8"
Sep 16 18:44:03.721346 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.721292    2554 scope.go:117] "RemoveContainer" containerID="e94b5f6d8f395e3cc186f62d7b07e7723de7e941befcdbdf99661306bad0c01f"
Sep 16 18:44:03.723022 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.722995    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"]
Sep 16 18:44:03.723983 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:44:03.723947    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94b5f6d8f395e3cc186f62d7b07e7723de7e941befcdbdf99661306bad0c01f\": container with ID starting with e94b5f6d8f395e3cc186f62d7b07e7723de7e941befcdbdf99661306bad0c01f not found: ID does not exist" containerID="e94b5f6d8f395e3cc186f62d7b07e7723de7e941befcdbdf99661306bad0c01f"
Sep 16 18:44:03.724086 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.723993    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94b5f6d8f395e3cc186f62d7b07e7723de7e941befcdbdf99661306bad0c01f"} err="failed to get container status \"e94b5f6d8f395e3cc186f62d7b07e7723de7e941befcdbdf99661306bad0c01f\": rpc error: code = NotFound desc = could not find container \"e94b5f6d8f395e3cc186f62d7b07e7723de7e941befcdbdf99661306bad0c01f\": container with ID starting with e94b5f6d8f395e3cc186f62d7b07e7723de7e941befcdbdf99661306bad0c01f not found: ID does not exist"
Sep 16 18:44:03.724086 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.724021    2554 scope.go:117] "RemoveContainer" containerID="03e3a6fe890ef504cc7cbc4f285680452e4e758d157e0dba5b81c90270a0d2a8"
Sep 16 18:44:03.724469 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:44:03.724442    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e3a6fe890ef504cc7cbc4f285680452e4e758d157e0dba5b81c90270a0d2a8\": container with ID starting with 03e3a6fe890ef504cc7cbc4f285680452e4e758d157e0dba5b81c90270a0d2a8 not found: ID does not exist" containerID="03e3a6fe890ef504cc7cbc4f285680452e4e758d157e0dba5b81c90270a0d2a8"
Sep 16 18:44:03.724561 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.724479    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e3a6fe890ef504cc7cbc4f285680452e4e758d157e0dba5b81c90270a0d2a8"} err="failed to get container status \"03e3a6fe890ef504cc7cbc4f285680452e4e758d157e0dba5b81c90270a0d2a8\": rpc error: code = NotFound desc = could not find container \"03e3a6fe890ef504cc7cbc4f285680452e4e758d157e0dba5b81c90270a0d2a8\": container with ID starting with 03e3a6fe890ef504cc7cbc4f285680452e4e758d157e0dba5b81c90270a0d2a8 not found: ID does not exist"
Sep 16 18:44:03.727474 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:03.727452    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6a7741-openshift-default-5ddbc78544-zsqj6"]
Sep 16 18:44:04.886274 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:04.886223    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="400fa802-1bf9-4020-90a6-639087dfafa7" path="/var/lib/kubelet/pods/400fa802-1bf9-4020-90a6-639087dfafa7/volumes"
Sep 16 18:44:14.075211 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.075176    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"]
Sep 16 18:44:14.075937 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.075881    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll" podUID="c6ebd759-e409-47b9-a680-60c080b6d96f" containerName="maas-api" containerID="cri-o://2b88dd3f19168e2039fb0c398c5e0f864f9ddbd7c03f59a99d04e0d9f7e84e08" gracePeriod=30
Sep 16 18:44:14.148627 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.146990    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"]
Sep 16 18:44:14.149029 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.148837    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r" podUID="5abcadd6-15bc-4bf1-8d39-7fa8162eba56" containerName="payload-pre-processing" containerID="cri-o://63e76f89712054ff6676825d7fd16665f615847c5229e81690193383fceb58a6" gracePeriod=30
Sep 16 18:44:14.417273 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.417243    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:44:14.449078 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.449048    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"
Sep 16 18:44:14.510783 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.510749    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5abcadd6-15bc-4bf1-8d39-7fa8162eba56-plugins-config-volume\") pod \"5abcadd6-15bc-4bf1-8d39-7fa8162eba56\" (UID: \"5abcadd6-15bc-4bf1-8d39-7fa8162eba56\") "
Sep 16 18:44:14.510783 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.510789    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6ebd759-e409-47b9-a680-60c080b6d96f-metrics-certs\") pod \"c6ebd759-e409-47b9-a680-60c080b6d96f\" (UID: \"c6ebd759-e409-47b9-a680-60c080b6d96f\") "
Sep 16 18:44:14.510973 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.510913    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v72d7\" (UniqueName: \"kubernetes.io/projected/5abcadd6-15bc-4bf1-8d39-7fa8162eba56-kube-api-access-v72d7\") pod \"5abcadd6-15bc-4bf1-8d39-7fa8162eba56\" (UID: \"5abcadd6-15bc-4bf1-8d39-7fa8162eba56\") "
Sep 16 18:44:14.510973 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.510951    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ebd759-e409-47b9-a680-60c080b6d96f-openshift-service-ca\") pod \"c6ebd759-e409-47b9-a680-60c080b6d96f\" (UID: \"c6ebd759-e409-47b9-a680-60c080b6d96f\") "
Sep 16 18:44:14.511049 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.511036    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ebd759-e409-47b9-a680-60c080b6d96f-maas-api-tls\") pod \"c6ebd759-e409-47b9-a680-60c080b6d96f\" (UID: \"c6ebd759-e409-47b9-a680-60c080b6d96f\") "
Sep 16 18:44:14.511139 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.511110    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgp5g\" (UniqueName: \"kubernetes.io/projected/c6ebd759-e409-47b9-a680-60c080b6d96f-kube-api-access-wgp5g\") pod \"c6ebd759-e409-47b9-a680-60c080b6d96f\" (UID: \"c6ebd759-e409-47b9-a680-60c080b6d96f\") "
Sep 16 18:44:14.511293 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.511143    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abcadd6-15bc-4bf1-8d39-7fa8162eba56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5abcadd6-15bc-4bf1-8d39-7fa8162eba56" (UID: "5abcadd6-15bc-4bf1-8d39-7fa8162eba56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:14.511409 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.511383    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ebd759-e409-47b9-a680-60c080b6d96f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c6ebd759-e409-47b9-a680-60c080b6d96f" (UID: "c6ebd759-e409-47b9-a680-60c080b6d96f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:14.511496 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.511466    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5abcadd6-15bc-4bf1-8d39-7fa8162eba56-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:14.511496 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.511486    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ebd759-e409-47b9-a680-60c080b6d96f-openshift-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:14.513016 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.512996    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abcadd6-15bc-4bf1-8d39-7fa8162eba56-kube-api-access-v72d7" (OuterVolumeSpecName: "kube-api-access-v72d7") pod "5abcadd6-15bc-4bf1-8d39-7fa8162eba56" (UID: "5abcadd6-15bc-4bf1-8d39-7fa8162eba56"). InnerVolumeSpecName "kube-api-access-v72d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:14.513106 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.513085    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ebd759-e409-47b9-a680-60c080b6d96f-kube-api-access-wgp5g" (OuterVolumeSpecName: "kube-api-access-wgp5g") pod "c6ebd759-e409-47b9-a680-60c080b6d96f" (UID: "c6ebd759-e409-47b9-a680-60c080b6d96f"). InnerVolumeSpecName "kube-api-access-wgp5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:14.513388 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.513369    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ebd759-e409-47b9-a680-60c080b6d96f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c6ebd759-e409-47b9-a680-60c080b6d96f" (UID: "c6ebd759-e409-47b9-a680-60c080b6d96f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:14.513438 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.513387    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ebd759-e409-47b9-a680-60c080b6d96f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c6ebd759-e409-47b9-a680-60c080b6d96f" (UID: "c6ebd759-e409-47b9-a680-60c080b6d96f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:14.612707 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.612605    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6ebd759-e409-47b9-a680-60c080b6d96f-maas-api-tls\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:14.612707 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.612650    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgp5g\" (UniqueName: \"kubernetes.io/projected/c6ebd759-e409-47b9-a680-60c080b6d96f-kube-api-access-wgp5g\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:14.612707 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.612660    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6ebd759-e409-47b9-a680-60c080b6d96f-metrics-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:14.612707 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.612669    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v72d7\" (UniqueName: \"kubernetes.io/projected/5abcadd6-15bc-4bf1-8d39-7fa8162eba56-kube-api-access-v72d7\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:14.760836 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.760805    2554 generic.go:356] "Generic (PLEG): container finished" podID="5abcadd6-15bc-4bf1-8d39-7fa8162eba56" containerID="63e76f89712054ff6676825d7fd16665f615847c5229e81690193383fceb58a6" exitCode=0
Sep 16 18:44:14.761030 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.760859    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"
Sep 16 18:44:14.761030 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.760873    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r" event={"ID":"5abcadd6-15bc-4bf1-8d39-7fa8162eba56","Type":"ContainerDied","Data":"63e76f89712054ff6676825d7fd16665f615847c5229e81690193383fceb58a6"}
Sep 16 18:44:14.761030 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.760916    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r" event={"ID":"5abcadd6-15bc-4bf1-8d39-7fa8162eba56","Type":"ContainerDied","Data":"33f7cf6cb067f5aff3a9cd3bb6e89988f910856a9c1b283c7048f07dc82ea779"}
Sep 16 18:44:14.761030 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.760936    2554 scope.go:117] "RemoveContainer" containerID="63e76f89712054ff6676825d7fd16665f615847c5229e81690193383fceb58a6"
Sep 16 18:44:14.762372 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.762342    2554 generic.go:356] "Generic (PLEG): container finished" podID="c6ebd759-e409-47b9-a680-60c080b6d96f" containerID="2b88dd3f19168e2039fb0c398c5e0f864f9ddbd7c03f59a99d04e0d9f7e84e08" exitCode=0
Sep 16 18:44:14.762466 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.762373    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll" event={"ID":"c6ebd759-e409-47b9-a680-60c080b6d96f","Type":"ContainerDied","Data":"2b88dd3f19168e2039fb0c398c5e0f864f9ddbd7c03f59a99d04e0d9f7e84e08"}
Sep 16 18:44:14.762466 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.762392    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"
Sep 16 18:44:14.762466 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.762421    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll" event={"ID":"c6ebd759-e409-47b9-a680-60c080b6d96f","Type":"ContainerDied","Data":"a8f88d414f92d4df65294846d50dfa9b0b6326e501d2bf402475ab024bc40b62"}
Sep 16 18:44:14.769898 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.769880    2554 scope.go:117] "RemoveContainer" containerID="63e76f89712054ff6676825d7fd16665f615847c5229e81690193383fceb58a6"
Sep 16 18:44:14.770183 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:44:14.770161    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e76f89712054ff6676825d7fd16665f615847c5229e81690193383fceb58a6\": container with ID starting with 63e76f89712054ff6676825d7fd16665f615847c5229e81690193383fceb58a6 not found: ID does not exist" containerID="63e76f89712054ff6676825d7fd16665f615847c5229e81690193383fceb58a6"
Sep 16 18:44:14.770284 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.770190    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e76f89712054ff6676825d7fd16665f615847c5229e81690193383fceb58a6"} err="failed to get container status \"63e76f89712054ff6676825d7fd16665f615847c5229e81690193383fceb58a6\": rpc error: code = NotFound desc = could not find container \"63e76f89712054ff6676825d7fd16665f615847c5229e81690193383fceb58a6\": container with ID starting with 63e76f89712054ff6676825d7fd16665f615847c5229e81690193383fceb58a6 not found: ID does not exist"
Sep 16 18:44:14.770284 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.770206    2554 scope.go:117] "RemoveContainer" containerID="2b88dd3f19168e2039fb0c398c5e0f864f9ddbd7c03f59a99d04e0d9f7e84e08"
Sep 16 18:44:14.779390 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.779371    2554 scope.go:117] "RemoveContainer" containerID="2b88dd3f19168e2039fb0c398c5e0f864f9ddbd7c03f59a99d04e0d9f7e84e08"
Sep 16 18:44:14.779682 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:44:14.779663    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b88dd3f19168e2039fb0c398c5e0f864f9ddbd7c03f59a99d04e0d9f7e84e08\": container with ID starting with 2b88dd3f19168e2039fb0c398c5e0f864f9ddbd7c03f59a99d04e0d9f7e84e08 not found: ID does not exist" containerID="2b88dd3f19168e2039fb0c398c5e0f864f9ddbd7c03f59a99d04e0d9f7e84e08"
Sep 16 18:44:14.779723 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.779689    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b88dd3f19168e2039fb0c398c5e0f864f9ddbd7c03f59a99d04e0d9f7e84e08"} err="failed to get container status \"2b88dd3f19168e2039fb0c398c5e0f864f9ddbd7c03f59a99d04e0d9f7e84e08\": rpc error: code = NotFound desc = could not find container \"2b88dd3f19168e2039fb0c398c5e0f864f9ddbd7c03f59a99d04e0d9f7e84e08\": container with ID starting with 2b88dd3f19168e2039fb0c398c5e0f864f9ddbd7c03f59a99d04e0d9f7e84e08 not found: ID does not exist"
Sep 16 18:44:14.787109 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.787084    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"]
Sep 16 18:44:14.793794 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.793771    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-61fa84-56b5f896f6-5svll"]
Sep 16 18:44:14.804167 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.804144    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"]
Sep 16 18:44:14.807963 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.807941    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-61fa84-f79ffbfdb-w9g4r"]
Sep 16 18:44:14.891410 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.891335    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5abcadd6-15bc-4bf1-8d39-7fa8162eba56" path="/var/lib/kubelet/pods/5abcadd6-15bc-4bf1-8d39-7fa8162eba56/volumes"
Sep 16 18:44:14.891833 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:14.891810    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6ebd759-e409-47b9-a680-60c080b6d96f" path="/var/lib/kubelet/pods/c6ebd759-e409-47b9-a680-60c080b6d96f/volumes"
Sep 16 18:44:19.703278 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:19.703242    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:44:32.470844 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.470810    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"]
Sep 16 18:44:32.471310 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.471042    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf" podUID="3ef3ad01-7823-466a-a3a4-274395e56a75" containerName="payload-processing" containerID="cri-o://857536a6bb015da6bda3434931423d0b232cee9d6f210506f99a65ec5863c844" gracePeriod=30
Sep 16 18:44:32.481805 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.481775    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"]
Sep 16 18:44:32.482045 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.482018    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25" podUID="e4c34855-8f87-4fca-85b6-ac474f5a8359" containerName="payload-pre-processing" containerID="cri-o://7321101d0b08c64c0e20e563580ef02a61373e4474984e1fc478dbaac503aaeb" gracePeriod=30
Sep 16 18:44:32.736395 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.736373    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"
Sep 16 18:44:32.772798 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.772771    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"
Sep 16 18:44:32.794619 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.794592    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef3ad01-7823-466a-a3a4-274395e56a75-plugins-config-volume\") pod \"3ef3ad01-7823-466a-a3a4-274395e56a75\" (UID: \"3ef3ad01-7823-466a-a3a4-274395e56a75\") "
Sep 16 18:44:32.794846 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.794652    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sh2z\" (UniqueName: \"kubernetes.io/projected/3ef3ad01-7823-466a-a3a4-274395e56a75-kube-api-access-2sh2z\") pod \"3ef3ad01-7823-466a-a3a4-274395e56a75\" (UID: \"3ef3ad01-7823-466a-a3a4-274395e56a75\") "
Sep 16 18:44:32.795052 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.795027    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef3ad01-7823-466a-a3a4-274395e56a75-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ef3ad01-7823-466a-a3a4-274395e56a75" (UID: "3ef3ad01-7823-466a-a3a4-274395e56a75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:32.796787 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.796759    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef3ad01-7823-466a-a3a4-274395e56a75-kube-api-access-2sh2z" (OuterVolumeSpecName: "kube-api-access-2sh2z") pod "3ef3ad01-7823-466a-a3a4-274395e56a75" (UID: "3ef3ad01-7823-466a-a3a4-274395e56a75"). InnerVolumeSpecName "kube-api-access-2sh2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:32.866100 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.866069    2554 generic.go:356] "Generic (PLEG): container finished" podID="e4c34855-8f87-4fca-85b6-ac474f5a8359" containerID="7321101d0b08c64c0e20e563580ef02a61373e4474984e1fc478dbaac503aaeb" exitCode=0
Sep 16 18:44:32.866283 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.866110    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"
Sep 16 18:44:32.866283 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.866152    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25" event={"ID":"e4c34855-8f87-4fca-85b6-ac474f5a8359","Type":"ContainerDied","Data":"7321101d0b08c64c0e20e563580ef02a61373e4474984e1fc478dbaac503aaeb"}
Sep 16 18:44:32.866283 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.866258    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25" event={"ID":"e4c34855-8f87-4fca-85b6-ac474f5a8359","Type":"ContainerDied","Data":"190830389e494e7bbd72ea479097a42e8963ca0500436959908466ce1b826db9"}
Sep 16 18:44:32.866504 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.866293    2554 scope.go:117] "RemoveContainer" containerID="7321101d0b08c64c0e20e563580ef02a61373e4474984e1fc478dbaac503aaeb"
Sep 16 18:44:32.867459 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.867436    2554 generic.go:356] "Generic (PLEG): container finished" podID="3ef3ad01-7823-466a-a3a4-274395e56a75" containerID="857536a6bb015da6bda3434931423d0b232cee9d6f210506f99a65ec5863c844" exitCode=0
Sep 16 18:44:32.867542 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.867475    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf" event={"ID":"3ef3ad01-7823-466a-a3a4-274395e56a75","Type":"ContainerDied","Data":"857536a6bb015da6bda3434931423d0b232cee9d6f210506f99a65ec5863c844"}
Sep 16 18:44:32.867542 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.867489    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"
Sep 16 18:44:32.867542 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.867516    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf" event={"ID":"3ef3ad01-7823-466a-a3a4-274395e56a75","Type":"ContainerDied","Data":"8d564758cd04c6ec884f45eff2f86018a0491646ff01a2c15817a9e29fdd8af1"}
Sep 16 18:44:32.875793 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.875778    2554 scope.go:117] "RemoveContainer" containerID="7321101d0b08c64c0e20e563580ef02a61373e4474984e1fc478dbaac503aaeb"
Sep 16 18:44:32.876060 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:44:32.876042    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7321101d0b08c64c0e20e563580ef02a61373e4474984e1fc478dbaac503aaeb\": container with ID starting with 7321101d0b08c64c0e20e563580ef02a61373e4474984e1fc478dbaac503aaeb not found: ID does not exist" containerID="7321101d0b08c64c0e20e563580ef02a61373e4474984e1fc478dbaac503aaeb"
Sep 16 18:44:32.876107 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.876068    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7321101d0b08c64c0e20e563580ef02a61373e4474984e1fc478dbaac503aaeb"} err="failed to get container status \"7321101d0b08c64c0e20e563580ef02a61373e4474984e1fc478dbaac503aaeb\": rpc error: code = NotFound desc = could not find container \"7321101d0b08c64c0e20e563580ef02a61373e4474984e1fc478dbaac503aaeb\": container with ID starting with 7321101d0b08c64c0e20e563580ef02a61373e4474984e1fc478dbaac503aaeb not found: ID does not exist"
Sep 16 18:44:32.876107 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.876084    2554 scope.go:117] "RemoveContainer" containerID="857536a6bb015da6bda3434931423d0b232cee9d6f210506f99a65ec5863c844"
Sep 16 18:44:32.884893 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.884875    2554 scope.go:117] "RemoveContainer" containerID="857536a6bb015da6bda3434931423d0b232cee9d6f210506f99a65ec5863c844"
Sep 16 18:44:32.885166 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:44:32.885149    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857536a6bb015da6bda3434931423d0b232cee9d6f210506f99a65ec5863c844\": container with ID starting with 857536a6bb015da6bda3434931423d0b232cee9d6f210506f99a65ec5863c844 not found: ID does not exist" containerID="857536a6bb015da6bda3434931423d0b232cee9d6f210506f99a65ec5863c844"
Sep 16 18:44:32.885213 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.885174    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857536a6bb015da6bda3434931423d0b232cee9d6f210506f99a65ec5863c844"} err="failed to get container status \"857536a6bb015da6bda3434931423d0b232cee9d6f210506f99a65ec5863c844\": rpc error: code = NotFound desc = could not find container \"857536a6bb015da6bda3434931423d0b232cee9d6f210506f99a65ec5863c844\": container with ID starting with 857536a6bb015da6bda3434931423d0b232cee9d6f210506f99a65ec5863c844 not found: ID does not exist"
Sep 16 18:44:32.895191 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.895171    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c34855-8f87-4fca-85b6-ac474f5a8359-plugins-config-volume\") pod \"e4c34855-8f87-4fca-85b6-ac474f5a8359\" (UID: \"e4c34855-8f87-4fca-85b6-ac474f5a8359\") "
Sep 16 18:44:32.895300 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.895282    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfl5x\" (UniqueName: \"kubernetes.io/projected/e4c34855-8f87-4fca-85b6-ac474f5a8359-kube-api-access-xfl5x\") pod \"e4c34855-8f87-4fca-85b6-ac474f5a8359\" (UID: \"e4c34855-8f87-4fca-85b6-ac474f5a8359\") "
Sep 16 18:44:32.895571 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.895546    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c34855-8f87-4fca-85b6-ac474f5a8359-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4c34855-8f87-4fca-85b6-ac474f5a8359" (UID: "e4c34855-8f87-4fca-85b6-ac474f5a8359"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:32.895630 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.895557    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef3ad01-7823-466a-a3a4-274395e56a75-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:32.895630 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.895605    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sh2z\" (UniqueName: \"kubernetes.io/projected/3ef3ad01-7823-466a-a3a4-274395e56a75-kube-api-access-2sh2z\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:32.897579 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.897555    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c34855-8f87-4fca-85b6-ac474f5a8359-kube-api-access-xfl5x" (OuterVolumeSpecName: "kube-api-access-xfl5x") pod "e4c34855-8f87-4fca-85b6-ac474f5a8359" (UID: "e4c34855-8f87-4fca-85b6-ac474f5a8359"). InnerVolumeSpecName "kube-api-access-xfl5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:32.898542 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.898523    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"]
Sep 16 18:44:32.921792 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.921769    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ad6826-7456f4945f-wlsgf"]
Sep 16 18:44:32.996415 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.996335    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c34855-8f87-4fca-85b6-ac474f5a8359-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:32.996415 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:32.996361    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfl5x\" (UniqueName: \"kubernetes.io/projected/e4c34855-8f87-4fca-85b6-ac474f5a8359-kube-api-access-xfl5x\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:33.193080 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:33.193047    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"]
Sep 16 18:44:33.209476 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:33.209435    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ad6826-84cfc8b94-9vt25"]
Sep 16 18:44:34.884096 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:34.884059    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ef3ad01-7823-466a-a3a4-274395e56a75" path="/var/lib/kubelet/pods/3ef3ad01-7823-466a-a3a4-274395e56a75/volumes"
Sep 16 18:44:34.884535 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:34.884509    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4c34855-8f87-4fca-85b6-ac474f5a8359" path="/var/lib/kubelet/pods/e4c34855-8f87-4fca-85b6-ac474f5a8359/volumes"
Sep 16 18:44:40.109502 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:40.109424    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"]
Sep 16 18:44:40.109900 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:40.109658    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4" podUID="94b2ef39-794f-4860-9a43-a3ba52fcb217" containerName="istio-proxy" containerID="cri-o://92fa280136b69c61f6b6b21ae97ffce70e9daadbf6587c4876a2f318697d8fdc" gracePeriod=30
Sep 16 18:44:44.009290 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:44.009254    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:44:45.366666 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.366642    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:44:45.526958 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.526921    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b2ef39-794f-4860-9a43-a3ba52fcb217-istiod-ca-cert\") pod \"94b2ef39-794f-4860-9a43-a3ba52fcb217\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") "
Sep 16 18:44:45.527139 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.526985    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-podinfo\") pod \"94b2ef39-794f-4860-9a43-a3ba52fcb217\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") "
Sep 16 18:44:45.527139 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.527003    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtl2c\" (UniqueName: \"kubernetes.io/projected/94b2ef39-794f-4860-9a43-a3ba52fcb217-kube-api-access-wtl2c\") pod \"94b2ef39-794f-4860-9a43-a3ba52fcb217\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") "
Sep 16 18:44:45.527265 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.527176    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-envoy\") pod \"94b2ef39-794f-4860-9a43-a3ba52fcb217\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") "
Sep 16 18:44:45.527342 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.527290    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-data\") pod \"94b2ef39-794f-4860-9a43-a3ba52fcb217\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") "
Sep 16 18:44:45.527342 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.527326    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-credential-socket\") pod \"94b2ef39-794f-4860-9a43-a3ba52fcb217\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") "
Sep 16 18:44:45.527447 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.527353    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b2ef39-794f-4860-9a43-a3ba52fcb217-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "94b2ef39-794f-4860-9a43-a3ba52fcb217" (UID: "94b2ef39-794f-4860-9a43-a3ba52fcb217"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:45.527447 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.527392    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-workload-socket\") pod \"94b2ef39-794f-4860-9a43-a3ba52fcb217\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") "
Sep 16 18:44:45.527447 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.527436    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-token\") pod \"94b2ef39-794f-4860-9a43-a3ba52fcb217\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") "
Sep 16 18:44:45.527613 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.527456    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-workload-certs\") pod \"94b2ef39-794f-4860-9a43-a3ba52fcb217\" (UID: \"94b2ef39-794f-4860-9a43-a3ba52fcb217\") "
Sep 16 18:44:45.527613 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.527521    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-data" (OuterVolumeSpecName: "istio-data") pod "94b2ef39-794f-4860-9a43-a3ba52fcb217" (UID: "94b2ef39-794f-4860-9a43-a3ba52fcb217"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:45.527714 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.527629    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "94b2ef39-794f-4860-9a43-a3ba52fcb217" (UID: "94b2ef39-794f-4860-9a43-a3ba52fcb217"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:45.527714 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.527665    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "94b2ef39-794f-4860-9a43-a3ba52fcb217" (UID: "94b2ef39-794f-4860-9a43-a3ba52fcb217"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:45.527826 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.527809    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "94b2ef39-794f-4860-9a43-a3ba52fcb217" (UID: "94b2ef39-794f-4860-9a43-a3ba52fcb217"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:45.527883 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.527819    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:45.527883 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.527851    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:45.527883 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.527864    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:45.527883 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.527876    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b2ef39-794f-4860-9a43-a3ba52fcb217-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:45.529382 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.529354    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "94b2ef39-794f-4860-9a43-a3ba52fcb217" (UID: "94b2ef39-794f-4860-9a43-a3ba52fcb217"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:45.529524 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.529510    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-token" (OuterVolumeSpecName: "istio-token") pod "94b2ef39-794f-4860-9a43-a3ba52fcb217" (UID: "94b2ef39-794f-4860-9a43-a3ba52fcb217"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:45.529606 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.529582    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b2ef39-794f-4860-9a43-a3ba52fcb217-kube-api-access-wtl2c" (OuterVolumeSpecName: "kube-api-access-wtl2c") pod "94b2ef39-794f-4860-9a43-a3ba52fcb217" (UID: "94b2ef39-794f-4860-9a43-a3ba52fcb217"). InnerVolumeSpecName "kube-api-access-wtl2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:45.529786 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.529745    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "94b2ef39-794f-4860-9a43-a3ba52fcb217" (UID: "94b2ef39-794f-4860-9a43-a3ba52fcb217"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:44:45.629226 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.629193    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:45.629226 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.629222    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:45.629226 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.629250    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:45.629226 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.629259    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtl2c\" (UniqueName: \"kubernetes.io/projected/94b2ef39-794f-4860-9a43-a3ba52fcb217-kube-api-access-wtl2c\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:45.629504 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.629268    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b2ef39-794f-4860-9a43-a3ba52fcb217-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:45.937767 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.937687    2554 generic.go:356] "Generic (PLEG): container finished" podID="94b2ef39-794f-4860-9a43-a3ba52fcb217" containerID="92fa280136b69c61f6b6b21ae97ffce70e9daadbf6587c4876a2f318697d8fdc" exitCode=0
Sep 16 18:44:45.937767 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.937734    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"
Sep 16 18:44:45.937949 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.937771    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4" event={"ID":"94b2ef39-794f-4860-9a43-a3ba52fcb217","Type":"ContainerDied","Data":"92fa280136b69c61f6b6b21ae97ffce70e9daadbf6587c4876a2f318697d8fdc"}
Sep 16 18:44:45.937949 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.937827    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4" event={"ID":"94b2ef39-794f-4860-9a43-a3ba52fcb217","Type":"ContainerDied","Data":"75eb9d27d8c9df09fc0ea757af72cb43b74662cc7fea0a946383a34fdf0e2c73"}
Sep 16 18:44:45.937949 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.937851    2554 scope.go:117] "RemoveContainer" containerID="92fa280136b69c61f6b6b21ae97ffce70e9daadbf6587c4876a2f318697d8fdc"
Sep 16 18:44:45.948307 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.948285    2554 scope.go:117] "RemoveContainer" containerID="d1f389915d6f95bcd9b153d5ff93685693cc719b21b09bcea4a9421006cd8dbd"
Sep 16 18:44:45.959044 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.959014    2554 scope.go:117] "RemoveContainer" containerID="92fa280136b69c61f6b6b21ae97ffce70e9daadbf6587c4876a2f318697d8fdc"
Sep 16 18:44:45.959724 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:44:45.959692    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92fa280136b69c61f6b6b21ae97ffce70e9daadbf6587c4876a2f318697d8fdc\": container with ID starting with 92fa280136b69c61f6b6b21ae97ffce70e9daadbf6587c4876a2f318697d8fdc not found: ID does not exist" containerID="92fa280136b69c61f6b6b21ae97ffce70e9daadbf6587c4876a2f318697d8fdc"
Sep 16 18:44:45.959798 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.959734    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fa280136b69c61f6b6b21ae97ffce70e9daadbf6587c4876a2f318697d8fdc"} err="failed to get container status \"92fa280136b69c61f6b6b21ae97ffce70e9daadbf6587c4876a2f318697d8fdc\": rpc error: code = NotFound desc = could not find container \"92fa280136b69c61f6b6b21ae97ffce70e9daadbf6587c4876a2f318697d8fdc\": container with ID starting with 92fa280136b69c61f6b6b21ae97ffce70e9daadbf6587c4876a2f318697d8fdc not found: ID does not exist"
Sep 16 18:44:45.959798 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.959758    2554 scope.go:117] "RemoveContainer" containerID="d1f389915d6f95bcd9b153d5ff93685693cc719b21b09bcea4a9421006cd8dbd"
Sep 16 18:44:45.960058 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:44:45.960040    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f389915d6f95bcd9b153d5ff93685693cc719b21b09bcea4a9421006cd8dbd\": container with ID starting with d1f389915d6f95bcd9b153d5ff93685693cc719b21b09bcea4a9421006cd8dbd not found: ID does not exist" containerID="d1f389915d6f95bcd9b153d5ff93685693cc719b21b09bcea4a9421006cd8dbd"
Sep 16 18:44:45.960116 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.960067    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f389915d6f95bcd9b153d5ff93685693cc719b21b09bcea4a9421006cd8dbd"} err="failed to get container status \"d1f389915d6f95bcd9b153d5ff93685693cc719b21b09bcea4a9421006cd8dbd\": rpc error: code = NotFound desc = could not find container \"d1f389915d6f95bcd9b153d5ff93685693cc719b21b09bcea4a9421006cd8dbd\": container with ID starting with d1f389915d6f95bcd9b153d5ff93685693cc719b21b09bcea4a9421006cd8dbd not found: ID does not exist"
Sep 16 18:44:45.964288 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.964265    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"]
Sep 16 18:44:45.970540 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:45.970518    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ad6826-openshift-default-6d4cd784bd-bkwf4"]
Sep 16 18:44:46.884428 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:46.884398    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94b2ef39-794f-4860-9a43-a3ba52fcb217" path="/var/lib/kubelet/pods/94b2ef39-794f-4860-9a43-a3ba52fcb217/volumes"
Sep 16 18:44:53.866006 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:53.865965    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"]
Sep 16 18:44:54.331440 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.331406    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"]
Sep 16 18:44:54.331692 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.331658    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv" podUID="655f67e1-9da5-4ccf-b57c-c085d8bb523e" containerName="payload-processing" containerID="cri-o://9b331d7f94c07f93eaabedf38127a7d8fdf9ae859ee3708fd96d277990f3710f" gracePeriod=30
Sep 16 18:44:54.338409 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.338331    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"]
Sep 16 18:44:54.338903 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.338534    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw" podUID="22f4dc05-42f8-4fc7-8543-15cefc4e80c7" containerName="payload-pre-processing" containerID="cri-o://a5a4a636c76a9b820ea72acbaf2ca4364f670a62dd1096cb7cf82005e947ecca" gracePeriod=30
Sep 16 18:44:54.628189 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.628155    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"
Sep 16 18:44:54.631016 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.630995    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"
Sep 16 18:44:54.709893 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.709865    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f4dc05-42f8-4fc7-8543-15cefc4e80c7-plugins-config-volume\") pod \"22f4dc05-42f8-4fc7-8543-15cefc4e80c7\" (UID: \"22f4dc05-42f8-4fc7-8543-15cefc4e80c7\") "
Sep 16 18:44:54.710067 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.709912    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd4jp\" (UniqueName: \"kubernetes.io/projected/655f67e1-9da5-4ccf-b57c-c085d8bb523e-kube-api-access-bd4jp\") pod \"655f67e1-9da5-4ccf-b57c-c085d8bb523e\" (UID: \"655f67e1-9da5-4ccf-b57c-c085d8bb523e\") "
Sep 16 18:44:54.710127 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.710089    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kszh\" (UniqueName: \"kubernetes.io/projected/22f4dc05-42f8-4fc7-8543-15cefc4e80c7-kube-api-access-2kszh\") pod \"22f4dc05-42f8-4fc7-8543-15cefc4e80c7\" (UID: \"22f4dc05-42f8-4fc7-8543-15cefc4e80c7\") "
Sep 16 18:44:54.710179 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.710161    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655f67e1-9da5-4ccf-b57c-c085d8bb523e-plugins-config-volume\") pod \"655f67e1-9da5-4ccf-b57c-c085d8bb523e\" (UID: \"655f67e1-9da5-4ccf-b57c-c085d8bb523e\") "
Sep 16 18:44:54.710325 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.710294    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f4dc05-42f8-4fc7-8543-15cefc4e80c7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22f4dc05-42f8-4fc7-8543-15cefc4e80c7" (UID: "22f4dc05-42f8-4fc7-8543-15cefc4e80c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:54.710504 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.710483    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f4dc05-42f8-4fc7-8543-15cefc4e80c7-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:54.710582 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.710545    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655f67e1-9da5-4ccf-b57c-c085d8bb523e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "655f67e1-9da5-4ccf-b57c-c085d8bb523e" (UID: "655f67e1-9da5-4ccf-b57c-c085d8bb523e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:54.712018 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.711992    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655f67e1-9da5-4ccf-b57c-c085d8bb523e-kube-api-access-bd4jp" (OuterVolumeSpecName: "kube-api-access-bd4jp") pod "655f67e1-9da5-4ccf-b57c-c085d8bb523e" (UID: "655f67e1-9da5-4ccf-b57c-c085d8bb523e"). InnerVolumeSpecName "kube-api-access-bd4jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:54.712109 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.712063    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f4dc05-42f8-4fc7-8543-15cefc4e80c7-kube-api-access-2kszh" (OuterVolumeSpecName: "kube-api-access-2kszh") pod "22f4dc05-42f8-4fc7-8543-15cefc4e80c7" (UID: "22f4dc05-42f8-4fc7-8543-15cefc4e80c7"). InnerVolumeSpecName "kube-api-access-2kszh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:54.811749 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.811718    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd4jp\" (UniqueName: \"kubernetes.io/projected/655f67e1-9da5-4ccf-b57c-c085d8bb523e-kube-api-access-bd4jp\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:54.811749 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.811744    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kszh\" (UniqueName: \"kubernetes.io/projected/22f4dc05-42f8-4fc7-8543-15cefc4e80c7-kube-api-access-2kszh\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:54.811749 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.811754    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655f67e1-9da5-4ccf-b57c-c085d8bb523e-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:44:54.989397 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.989365    2554 generic.go:356] "Generic (PLEG): container finished" podID="655f67e1-9da5-4ccf-b57c-c085d8bb523e" containerID="9b331d7f94c07f93eaabedf38127a7d8fdf9ae859ee3708fd96d277990f3710f" exitCode=0
Sep 16 18:44:54.989855 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.989413    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"
Sep 16 18:44:54.989855 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.989450    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv" event={"ID":"655f67e1-9da5-4ccf-b57c-c085d8bb523e","Type":"ContainerDied","Data":"9b331d7f94c07f93eaabedf38127a7d8fdf9ae859ee3708fd96d277990f3710f"}
Sep 16 18:44:54.989855 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.989500    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv" event={"ID":"655f67e1-9da5-4ccf-b57c-c085d8bb523e","Type":"ContainerDied","Data":"0d1a3cc830084b56cd32e05f6045bfbe15663eb36a2f351620f3e6bf64dc0206"}
Sep 16 18:44:54.989855 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.989528    2554 scope.go:117] "RemoveContainer" containerID="9b331d7f94c07f93eaabedf38127a7d8fdf9ae859ee3708fd96d277990f3710f"
Sep 16 18:44:54.990957 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.990933    2554 generic.go:356] "Generic (PLEG): container finished" podID="22f4dc05-42f8-4fc7-8543-15cefc4e80c7" containerID="a5a4a636c76a9b820ea72acbaf2ca4364f670a62dd1096cb7cf82005e947ecca" exitCode=0
Sep 16 18:44:54.991087 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.990974    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"
Sep 16 18:44:54.991087 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.990990    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw" event={"ID":"22f4dc05-42f8-4fc7-8543-15cefc4e80c7","Type":"ContainerDied","Data":"a5a4a636c76a9b820ea72acbaf2ca4364f670a62dd1096cb7cf82005e947ecca"}
Sep 16 18:44:54.991087 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.991038    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw" event={"ID":"22f4dc05-42f8-4fc7-8543-15cefc4e80c7","Type":"ContainerDied","Data":"7e7cb6f5cda9b81cbe6835e3693c961b7ac86c292d6a4d809bf116a68d76936a"}
Sep 16 18:44:54.998408 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.998393    2554 scope.go:117] "RemoveContainer" containerID="9b331d7f94c07f93eaabedf38127a7d8fdf9ae859ee3708fd96d277990f3710f"
Sep 16 18:44:54.998681 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:44:54.998661    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b331d7f94c07f93eaabedf38127a7d8fdf9ae859ee3708fd96d277990f3710f\": container with ID starting with 9b331d7f94c07f93eaabedf38127a7d8fdf9ae859ee3708fd96d277990f3710f not found: ID does not exist" containerID="9b331d7f94c07f93eaabedf38127a7d8fdf9ae859ee3708fd96d277990f3710f"
Sep 16 18:44:54.998741 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.998690    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b331d7f94c07f93eaabedf38127a7d8fdf9ae859ee3708fd96d277990f3710f"} err="failed to get container status \"9b331d7f94c07f93eaabedf38127a7d8fdf9ae859ee3708fd96d277990f3710f\": rpc error: code = NotFound desc = could not find container \"9b331d7f94c07f93eaabedf38127a7d8fdf9ae859ee3708fd96d277990f3710f\": container with ID starting with 9b331d7f94c07f93eaabedf38127a7d8fdf9ae859ee3708fd96d277990f3710f not found: ID does not exist"
Sep 16 18:44:54.998741 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:54.998704    2554 scope.go:117] "RemoveContainer" containerID="a5a4a636c76a9b820ea72acbaf2ca4364f670a62dd1096cb7cf82005e947ecca"
Sep 16 18:44:55.006619 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:55.006479    2554 scope.go:117] "RemoveContainer" containerID="a5a4a636c76a9b820ea72acbaf2ca4364f670a62dd1096cb7cf82005e947ecca"
Sep 16 18:44:55.006759 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:44:55.006741    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a4a636c76a9b820ea72acbaf2ca4364f670a62dd1096cb7cf82005e947ecca\": container with ID starting with a5a4a636c76a9b820ea72acbaf2ca4364f670a62dd1096cb7cf82005e947ecca not found: ID does not exist" containerID="a5a4a636c76a9b820ea72acbaf2ca4364f670a62dd1096cb7cf82005e947ecca"
Sep 16 18:44:55.006807 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:55.006765    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a4a636c76a9b820ea72acbaf2ca4364f670a62dd1096cb7cf82005e947ecca"} err="failed to get container status \"a5a4a636c76a9b820ea72acbaf2ca4364f670a62dd1096cb7cf82005e947ecca\": rpc error: code = NotFound desc = could not find container \"a5a4a636c76a9b820ea72acbaf2ca4364f670a62dd1096cb7cf82005e947ecca\": container with ID starting with a5a4a636c76a9b820ea72acbaf2ca4364f670a62dd1096cb7cf82005e947ecca not found: ID does not exist"
Sep 16 18:44:55.008039 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:55.008022    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"]
Sep 16 18:44:55.014935 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:55.014906    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cf62d1-57f65fc9f9-8w8gv"]
Sep 16 18:44:55.023381 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:55.023359    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"]
Sep 16 18:44:55.029450 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:55.029419    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cf62d1-5db57f6f58-j8dhw"]
Sep 16 18:44:56.883988 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:56.883955    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22f4dc05-42f8-4fc7-8543-15cefc4e80c7" path="/var/lib/kubelet/pods/22f4dc05-42f8-4fc7-8543-15cefc4e80c7/volumes"
Sep 16 18:44:56.884380 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:44:56.884347    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="655f67e1-9da5-4ccf-b57c-c085d8bb523e" path="/var/lib/kubelet/pods/655f67e1-9da5-4ccf-b57c-c085d8bb523e/volumes"
Sep 16 18:45:00.165626 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.165592    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d3e8be-29826405-lm4nq"]
Sep 16 18:45:00.166019 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.165965    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a4890a-36b1-4233-9e3d-e50cd10a2f21" containerName="payload-pre-processing"
Sep 16 18:45:00.166019 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.165975    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a4890a-36b1-4233-9e3d-e50cd10a2f21" containerName="payload-pre-processing"
Sep 16 18:45:00.166019 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.165984    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b2ef39-794f-4860-9a43-a3ba52fcb217" containerName="istio-proxy"
Sep 16 18:45:00.166019 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.165990    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b2ef39-794f-4860-9a43-a3ba52fcb217" containerName="istio-proxy"
Sep 16 18:45:00.166019 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166001    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f4dc05-42f8-4fc7-8543-15cefc4e80c7" containerName="payload-pre-processing"
Sep 16 18:45:00.166019 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166007    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f4dc05-42f8-4fc7-8543-15cefc4e80c7" containerName="payload-pre-processing"
Sep 16 18:45:00.166019 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166013    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="655f67e1-9da5-4ccf-b57c-c085d8bb523e" containerName="payload-processing"
Sep 16 18:45:00.166019 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166018    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="655f67e1-9da5-4ccf-b57c-c085d8bb523e" containerName="payload-processing"
Sep 16 18:45:00.166019 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166025    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ef3ad01-7823-466a-a3a4-274395e56a75" containerName="payload-processing"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166031    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ef3ad01-7823-466a-a3a4-274395e56a75" containerName="payload-processing"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166040    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5abcadd6-15bc-4bf1-8d39-7fa8162eba56" containerName="payload-pre-processing"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166045    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5abcadd6-15bc-4bf1-8d39-7fa8162eba56" containerName="payload-pre-processing"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166052    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400fa802-1bf9-4020-90a6-639087dfafa7" containerName="istio-proxy"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166057    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="400fa802-1bf9-4020-90a6-639087dfafa7" containerName="istio-proxy"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166063    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6ebd759-e409-47b9-a680-60c080b6d96f" containerName="maas-api"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166068    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6ebd759-e409-47b9-a680-60c080b6d96f" containerName="maas-api"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166081    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32667ec6-64cb-4b8f-87a2-87e7924794ed" containerName="payload-processing"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166085    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="32667ec6-64cb-4b8f-87a2-87e7924794ed" containerName="payload-processing"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166091    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee55eb5-c320-4715-b2a9-6c8b6ed4d800" containerName="main"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166095    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee55eb5-c320-4715-b2a9-6c8b6ed4d800" containerName="main"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166102    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c34855-8f87-4fca-85b6-ac474f5a8359" containerName="payload-pre-processing"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166108    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c34855-8f87-4fca-85b6-ac474f5a8359" containerName="payload-pre-processing"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166115    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335a6bc4-c505-4fab-b12f-d2ccc04e971d" containerName="istio-proxy"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166120    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="335a6bc4-c505-4fab-b12f-d2ccc04e971d" containerName="istio-proxy"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166125    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335a6bc4-c505-4fab-b12f-d2ccc04e971d" containerName="istio-proxy"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166130    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="335a6bc4-c505-4fab-b12f-d2ccc04e971d" containerName="istio-proxy"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166185    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="70a4890a-36b1-4233-9e3d-e50cd10a2f21" containerName="payload-pre-processing"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166193    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5abcadd6-15bc-4bf1-8d39-7fa8162eba56" containerName="payload-pre-processing"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166199    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="400fa802-1bf9-4020-90a6-639087dfafa7" containerName="istio-proxy"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166205    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c34855-8f87-4fca-85b6-ac474f5a8359" containerName="payload-pre-processing"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166210    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b2ef39-794f-4860-9a43-a3ba52fcb217" containerName="istio-proxy"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166215    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="32667ec6-64cb-4b8f-87a2-87e7924794ed" containerName="payload-processing"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166221    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="335a6bc4-c505-4fab-b12f-d2ccc04e971d" containerName="istio-proxy"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166225    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bee55eb5-c320-4715-b2a9-6c8b6ed4d800" containerName="main"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166242    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="335a6bc4-c505-4fab-b12f-d2ccc04e971d" containerName="istio-proxy"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166248    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ef3ad01-7823-466a-a3a4-274395e56a75" containerName="payload-processing"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166253    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="655f67e1-9da5-4ccf-b57c-c085d8bb523e" containerName="payload-processing"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166259    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f4dc05-42f8-4fc7-8543-15cefc4e80c7" containerName="payload-pre-processing"
Sep 16 18:45:00.166373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.166264    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6ebd759-e409-47b9-a680-60c080b6d96f" containerName="maas-api"
Sep 16 18:45:00.170879 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.170857    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d3e8be-29826405-lm4nq"
Sep 16 18:45:00.174528 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.174504    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lfw6p\""
Sep 16 18:45:00.180398 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.180381    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d3e8be-29826405-lm4nq"
Sep 16 18:45:00.196743 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.196638    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d3e8be-29826405-lm4nq"]
Sep 16 18:45:00.475072 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:00.475045    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d3e8be-29826405-lm4nq"]
Sep 16 18:45:00.477767 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:45:00.477739    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491a823f_3e5c_480a_98e0_6c40fd6cd562.slice/crio-f16f8ab8c7baa004acfb5d02ac959866240884f20eb410aa2cad6a408eb19579 WatchSource:0}: Error finding container f16f8ab8c7baa004acfb5d02ac959866240884f20eb410aa2cad6a408eb19579: Status 404 returned error can't find the container with id f16f8ab8c7baa004acfb5d02ac959866240884f20eb410aa2cad6a408eb19579
Sep 16 18:45:01.024899 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:01.024862    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d3e8be-29826405-lm4nq" event={"ID":"491a823f-3e5c-480a-98e0-6c40fd6cd562","Type":"ContainerStarted","Data":"fca3c4c41c5832921ceecabe326a80a85a78dc4586c38d04cb643618f61b5c76"}
Sep 16 18:45:01.025065 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:01.024925    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d3e8be-29826405-lm4nq" event={"ID":"491a823f-3e5c-480a-98e0-6c40fd6cd562","Type":"ContainerStarted","Data":"f16f8ab8c7baa004acfb5d02ac959866240884f20eb410aa2cad6a408eb19579"}
Sep 16 18:45:01.040412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:01.040351    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d3e8be-29826405-lm4nq" podStartSLOduration=1.04032876 podStartE2EDuration="1.04032876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:01.039727406 +0000 UTC m=+1724.598895237" watchObservedRunningTime="2026-09-16 18:45:01.04032876 +0000 UTC m=+1724.599496595"
Sep 16 18:45:02.032127 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:02.032092    2554 generic.go:356] "Generic (PLEG): container finished" podID="491a823f-3e5c-480a-98e0-6c40fd6cd562" containerID="fca3c4c41c5832921ceecabe326a80a85a78dc4586c38d04cb643618f61b5c76" exitCode=0
Sep 16 18:45:02.032541 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:02.032181    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d3e8be-29826405-lm4nq" event={"ID":"491a823f-3e5c-480a-98e0-6c40fd6cd562","Type":"ContainerDied","Data":"fca3c4c41c5832921ceecabe326a80a85a78dc4586c38d04cb643618f61b5c76"}
Sep 16 18:45:03.174675 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:03.174653    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d3e8be-29826405-lm4nq"
Sep 16 18:45:04.045878 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:04.045844    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d3e8be-29826405-lm4nq"
Sep 16 18:45:04.045878 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:04.045871    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d3e8be-29826405-lm4nq" event={"ID":"491a823f-3e5c-480a-98e0-6c40fd6cd562","Type":"ContainerDied","Data":"f16f8ab8c7baa004acfb5d02ac959866240884f20eb410aa2cad6a408eb19579"}
Sep 16 18:45:04.046090 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:04.045917    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f16f8ab8c7baa004acfb5d02ac959866240884f20eb410aa2cad6a408eb19579"
Sep 16 18:45:04.700671 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:04.700629    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:45:05.079760 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:05.079727    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"]
Sep 16 18:45:05.079981 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:05.079958    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c" podUID="e8b9a6dd-c92f-4782-973b-88d41b05bac7" containerName="maas-api" containerID="cri-o://9aa9807cb05c67376f78de6db0404a32ac40896640e25ce787b2060f922347b0" gracePeriod=30
Sep 16 18:45:05.326801 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:05.326774    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:45:05.401661 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:05.401573    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8b9a6dd-c92f-4782-973b-88d41b05bac7-maas-api-tls\") pod \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\" (UID: \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\") "
Sep 16 18:45:05.401661 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:05.401611    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8b9a6dd-c92f-4782-973b-88d41b05bac7-metrics-certs\") pod \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\" (UID: \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\") "
Sep 16 18:45:05.401661 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:05.401651    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwwrv\" (UniqueName: \"kubernetes.io/projected/e8b9a6dd-c92f-4782-973b-88d41b05bac7-kube-api-access-nwwrv\") pod \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\" (UID: \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\") "
Sep 16 18:45:05.401953 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:05.401683    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b9a6dd-c92f-4782-973b-88d41b05bac7-openshift-service-ca\") pod \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\" (UID: \"e8b9a6dd-c92f-4782-973b-88d41b05bac7\") "
Sep 16 18:45:05.402108 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:05.402079    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b9a6dd-c92f-4782-973b-88d41b05bac7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e8b9a6dd-c92f-4782-973b-88d41b05bac7" (UID: "e8b9a6dd-c92f-4782-973b-88d41b05bac7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:05.403726 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:05.403699    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b9a6dd-c92f-4782-973b-88d41b05bac7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e8b9a6dd-c92f-4782-973b-88d41b05bac7" (UID: "e8b9a6dd-c92f-4782-973b-88d41b05bac7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:05.404095 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:05.404072    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b9a6dd-c92f-4782-973b-88d41b05bac7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e8b9a6dd-c92f-4782-973b-88d41b05bac7" (UID: "e8b9a6dd-c92f-4782-973b-88d41b05bac7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:05.404095 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:05.404085    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b9a6dd-c92f-4782-973b-88d41b05bac7-kube-api-access-nwwrv" (OuterVolumeSpecName: "kube-api-access-nwwrv") pod "e8b9a6dd-c92f-4782-973b-88d41b05bac7" (UID: "e8b9a6dd-c92f-4782-973b-88d41b05bac7"). InnerVolumeSpecName "kube-api-access-nwwrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:05.502823 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:05.502789    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8b9a6dd-c92f-4782-973b-88d41b05bac7-metrics-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:45:05.502823 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:05.502817    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwwrv\" (UniqueName: \"kubernetes.io/projected/e8b9a6dd-c92f-4782-973b-88d41b05bac7-kube-api-access-nwwrv\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:45:05.502823 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:05.502827    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b9a6dd-c92f-4782-973b-88d41b05bac7-openshift-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:45:05.503042 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:05.502835    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8b9a6dd-c92f-4782-973b-88d41b05bac7-maas-api-tls\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:45:06.059206 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:06.059170    2554 generic.go:356] "Generic (PLEG): container finished" podID="e8b9a6dd-c92f-4782-973b-88d41b05bac7" containerID="9aa9807cb05c67376f78de6db0404a32ac40896640e25ce787b2060f922347b0" exitCode=0
Sep 16 18:45:06.059576 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:06.059219    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"
Sep 16 18:45:06.059576 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:06.059263    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c" event={"ID":"e8b9a6dd-c92f-4782-973b-88d41b05bac7","Type":"ContainerDied","Data":"9aa9807cb05c67376f78de6db0404a32ac40896640e25ce787b2060f922347b0"}
Sep 16 18:45:06.059576 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:06.059320    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c" event={"ID":"e8b9a6dd-c92f-4782-973b-88d41b05bac7","Type":"ContainerDied","Data":"0b9a45b2aacce3cd36d5f839ef01a58588affb708b639f7cac6cf505a912c7af"}
Sep 16 18:45:06.059576 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:06.059349    2554 scope.go:117] "RemoveContainer" containerID="9aa9807cb05c67376f78de6db0404a32ac40896640e25ce787b2060f922347b0"
Sep 16 18:45:06.068282 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:06.068266    2554 scope.go:117] "RemoveContainer" containerID="9aa9807cb05c67376f78de6db0404a32ac40896640e25ce787b2060f922347b0"
Sep 16 18:45:06.068561 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:45:06.068542    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa9807cb05c67376f78de6db0404a32ac40896640e25ce787b2060f922347b0\": container with ID starting with 9aa9807cb05c67376f78de6db0404a32ac40896640e25ce787b2060f922347b0 not found: ID does not exist" containerID="9aa9807cb05c67376f78de6db0404a32ac40896640e25ce787b2060f922347b0"
Sep 16 18:45:06.068602 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:06.068569    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa9807cb05c67376f78de6db0404a32ac40896640e25ce787b2060f922347b0"} err="failed to get container status \"9aa9807cb05c67376f78de6db0404a32ac40896640e25ce787b2060f922347b0\": rpc error: code = NotFound desc = could not find container \"9aa9807cb05c67376f78de6db0404a32ac40896640e25ce787b2060f922347b0\": container with ID starting with 9aa9807cb05c67376f78de6db0404a32ac40896640e25ce787b2060f922347b0 not found: ID does not exist"
Sep 16 18:45:06.084264 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:06.081118    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"]
Sep 16 18:45:06.087354 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:06.086967    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-6942a4-7969b88c4c-t2t9c"]
Sep 16 18:45:06.883815 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:06.883780    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8b9a6dd-c92f-4782-973b-88d41b05bac7" path="/var/lib/kubelet/pods/e8b9a6dd-c92f-4782-973b-88d41b05bac7/volumes"
Sep 16 18:45:08.884995 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:08.884946    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d" podUID="66b286d1-8e74-4b75-87ee-5d8213480721" containerName="main" containerID="cri-o://bdb5a2c9f5d3a0020ac8c46cb4d5dcec484889f9293816f4aa720bbdb4691017" gracePeriod=45
Sep 16 18:45:09.078217 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.078184    2554 generic.go:356] "Generic (PLEG): container finished" podID="66b286d1-8e74-4b75-87ee-5d8213480721" containerID="bdb5a2c9f5d3a0020ac8c46cb4d5dcec484889f9293816f4aa720bbdb4691017" exitCode=0
Sep 16 18:45:09.078360 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.078251    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d" event={"ID":"66b286d1-8e74-4b75-87ee-5d8213480721","Type":"ContainerDied","Data":"bdb5a2c9f5d3a0020ac8c46cb4d5dcec484889f9293816f4aa720bbdb4691017"}
Sep 16 18:45:09.136094 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.136036    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:45:09.234536 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.234504    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zpj7\" (UniqueName: \"kubernetes.io/projected/66b286d1-8e74-4b75-87ee-5d8213480721-kube-api-access-9zpj7\") pod \"66b286d1-8e74-4b75-87ee-5d8213480721\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") "
Sep 16 18:45:09.234725 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.234553    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-tmp-dir\") pod \"66b286d1-8e74-4b75-87ee-5d8213480721\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") "
Sep 16 18:45:09.234725 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.234581    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-model-cache\") pod \"66b286d1-8e74-4b75-87ee-5d8213480721\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") "
Sep 16 18:45:09.234725 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.234628    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66b286d1-8e74-4b75-87ee-5d8213480721-tls-certs\") pod \"66b286d1-8e74-4b75-87ee-5d8213480721\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") "
Sep 16 18:45:09.234725 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.234664    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-home\") pod \"66b286d1-8e74-4b75-87ee-5d8213480721\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") "
Sep 16 18:45:09.234725 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.234695    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-dshm\") pod \"66b286d1-8e74-4b75-87ee-5d8213480721\" (UID: \"66b286d1-8e74-4b75-87ee-5d8213480721\") "
Sep 16 18:45:09.234977 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.234840    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-model-cache" (OuterVolumeSpecName: "model-cache") pod "66b286d1-8e74-4b75-87ee-5d8213480721" (UID: "66b286d1-8e74-4b75-87ee-5d8213480721"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:09.235032 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.234997    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-model-cache\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:45:09.235187 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.235166    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-home" (OuterVolumeSpecName: "home") pod "66b286d1-8e74-4b75-87ee-5d8213480721" (UID: "66b286d1-8e74-4b75-87ee-5d8213480721"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:09.235370 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.235349    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66b286d1-8e74-4b75-87ee-5d8213480721" (UID: "66b286d1-8e74-4b75-87ee-5d8213480721"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:09.237192 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.237156    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b286d1-8e74-4b75-87ee-5d8213480721-kube-api-access-9zpj7" (OuterVolumeSpecName: "kube-api-access-9zpj7") pod "66b286d1-8e74-4b75-87ee-5d8213480721" (UID: "66b286d1-8e74-4b75-87ee-5d8213480721"). InnerVolumeSpecName "kube-api-access-9zpj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:09.237385 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.237362    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b286d1-8e74-4b75-87ee-5d8213480721-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66b286d1-8e74-4b75-87ee-5d8213480721" (UID: "66b286d1-8e74-4b75-87ee-5d8213480721"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:09.237454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.237377    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-dshm" (OuterVolumeSpecName: "dshm") pod "66b286d1-8e74-4b75-87ee-5d8213480721" (UID: "66b286d1-8e74-4b75-87ee-5d8213480721"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:09.336367 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.336329    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66b286d1-8e74-4b75-87ee-5d8213480721-tls-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:45:09.336367 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.336358    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-home\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:45:09.336367 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.336370    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-dshm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:45:09.336614 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.336383    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zpj7\" (UniqueName: \"kubernetes.io/projected/66b286d1-8e74-4b75-87ee-5d8213480721-kube-api-access-9zpj7\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:45:09.336614 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:09.336413    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66b286d1-8e74-4b75-87ee-5d8213480721-tmp-dir\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:45:10.084810 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:10.084781    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"
Sep 16 18:45:10.085259 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:10.084818    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d" event={"ID":"66b286d1-8e74-4b75-87ee-5d8213480721","Type":"ContainerDied","Data":"54ac1f0654676b01aed3419011123b276785a713bb06937bfa37409f40cfe87c"}
Sep 16 18:45:10.085259 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:10.084878    2554 scope.go:117] "RemoveContainer" containerID="bdb5a2c9f5d3a0020ac8c46cb4d5dcec484889f9293816f4aa720bbdb4691017"
Sep 16 18:45:10.105826 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:10.105786    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"]
Sep 16 18:45:10.111651 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:10.111625    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6942a4/distinct-2-w4-6942a4-kserve-5cd9ff649d-5bm5d"]
Sep 16 18:45:10.883739 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:10.883694    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66b286d1-8e74-4b75-87ee-5d8213480721" path="/var/lib/kubelet/pods/66b286d1-8e74-4b75-87ee-5d8213480721/volumes"
Sep 16 18:45:26.918979 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:26.918942    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:45:39.701503 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:39.701463    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:45:48.901324 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:48.901293    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"]
Sep 16 18:45:51.907844 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:51.907808    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:45:59.364751 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.364710    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"]
Sep 16 18:45:59.365347 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.365317    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66b286d1-8e74-4b75-87ee-5d8213480721" containerName="main"
Sep 16 18:45:59.365347 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.365336    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="66b286d1-8e74-4b75-87ee-5d8213480721" containerName="main"
Sep 16 18:45:59.365477 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.365353    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400fa802-1bf9-4020-90a6-639087dfafa7" containerName="istio-proxy"
Sep 16 18:45:59.365477 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.365361    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="400fa802-1bf9-4020-90a6-639087dfafa7" containerName="istio-proxy"
Sep 16 18:45:59.365477 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.365383    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b9a6dd-c92f-4782-973b-88d41b05bac7" containerName="maas-api"
Sep 16 18:45:59.365477 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.365392    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b9a6dd-c92f-4782-973b-88d41b05bac7" containerName="maas-api"
Sep 16 18:45:59.365477 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.365411    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="491a823f-3e5c-480a-98e0-6c40fd6cd562" containerName="cleanup"
Sep 16 18:45:59.365477 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.365422    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="491a823f-3e5c-480a-98e0-6c40fd6cd562" containerName="cleanup"
Sep 16 18:45:59.365477 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.365438    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b2ef39-794f-4860-9a43-a3ba52fcb217" containerName="istio-proxy"
Sep 16 18:45:59.365477 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.365447    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b2ef39-794f-4860-9a43-a3ba52fcb217" containerName="istio-proxy"
Sep 16 18:45:59.368298 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.366358    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="66b286d1-8e74-4b75-87ee-5d8213480721" containerName="main"
Sep 16 18:45:59.368298 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.366417    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8b9a6dd-c92f-4782-973b-88d41b05bac7" containerName="maas-api"
Sep 16 18:45:59.368298 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.366445    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="400fa802-1bf9-4020-90a6-639087dfafa7" containerName="istio-proxy"
Sep 16 18:45:59.368298 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.366456    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b2ef39-794f-4860-9a43-a3ba52fcb217" containerName="istio-proxy"
Sep 16 18:45:59.368298 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.366475    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="491a823f-3e5c-480a-98e0-6c40fd6cd562" containerName="cleanup"
Sep 16 18:45:59.372524 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.372500    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.375108 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.375088    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-972df0\"/\"sub-test-model-972df0-kserve-self-signed-certs\""
Sep 16 18:45:59.381305 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.381284    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"]
Sep 16 18:45:59.495073 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.495036    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-dshm\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.495073 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.495073    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd89922-74c9-4749-a1c2-4ea00408957f-tls-certs\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.495308 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.495146    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m92dj\" (UniqueName: \"kubernetes.io/projected/fdd89922-74c9-4749-a1c2-4ea00408957f-kube-api-access-m92dj\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.495308 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.495213    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-home\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.495308 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.495260    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-model-cache\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.495308 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.495293    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-tmp-dir\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.596356 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.596316    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-home\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.596356 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.596357    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-model-cache\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.596570 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.596480    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-tmp-dir\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.596570 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.596557    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-dshm\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.596646 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.596582    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd89922-74c9-4749-a1c2-4ea00408957f-tls-certs\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.596750 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.596724    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m92dj\" (UniqueName: \"kubernetes.io/projected/fdd89922-74c9-4749-a1c2-4ea00408957f-kube-api-access-m92dj\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.596828 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.596784    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-home\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.596881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.596821    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-model-cache\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.596881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.596866    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-tmp-dir\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.598834 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.598813    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-dshm\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.599035 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.599019    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd89922-74c9-4749-a1c2-4ea00408957f-tls-certs\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.605130 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.605102    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m92dj\" (UniqueName: \"kubernetes.io/projected/fdd89922-74c9-4749-a1c2-4ea00408957f-kube-api-access-m92dj\") pod \"sub-test-model-972df0-kserve-59f8b55475-lqc94\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") " pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.682988 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.682895    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:45:59.956469 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:45:59.956444    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"]
Sep 16 18:45:59.958589 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:45:59.958562    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdd89922_74c9_4749_a1c2_4ea00408957f.slice/crio-005c99788a9975c762825636537c78d1c8eb809701c59a256ca2a7e362e5142b WatchSource:0}: Error finding container 005c99788a9975c762825636537c78d1c8eb809701c59a256ca2a7e362e5142b: Status 404 returned error can't find the container with id 005c99788a9975c762825636537c78d1c8eb809701c59a256ca2a7e362e5142b
Sep 16 18:46:00.338480 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:00.338441    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94" event={"ID":"fdd89922-74c9-4749-a1c2-4ea00408957f","Type":"ContainerStarted","Data":"beb0960bbd705b1211de4939eca1fd4596e84102217013c2fe03281e4191e72f"}
Sep 16 18:46:00.338657 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:00.338489    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94" event={"ID":"fdd89922-74c9-4749-a1c2-4ea00408957f","Type":"ContainerStarted","Data":"005c99788a9975c762825636537c78d1c8eb809701c59a256ca2a7e362e5142b"}
Sep 16 18:46:00.372652 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:00.372594    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94" podStartSLOduration=1.3725788620000001 podStartE2EDuration="1.372578862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:00.371137074 +0000 UTC m=+1783.930304906" watchObservedRunningTime="2026-09-16 18:46:00.372578862 +0000 UTC m=+1783.931746695"
Sep 16 18:46:03.919493 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:03.919451    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr" podUID="0326dcc8-4daf-4971-bea8-7fb756f5b7fa" containerName="main" containerID="cri-o://0c6a0e00c117947bcfbaf6e04ab1526866803e038df723962eb6b2edeb745a22" gracePeriod=45
Sep 16 18:46:03.935950 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:03.935923    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:46:04.176695 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.176637    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:46:04.241916 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.241886    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-model-cache\") pod \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") "
Sep 16 18:46:04.242100 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.241936    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-home\") pod \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") "
Sep 16 18:46:04.242100 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.241975    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-tmp-dir\") pod \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") "
Sep 16 18:46:04.242100 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.242014    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-tls-certs\") pod \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") "
Sep 16 18:46:04.242100 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.242078    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fhcv\" (UniqueName: \"kubernetes.io/projected/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-kube-api-access-5fhcv\") pod \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") "
Sep 16 18:46:04.242347 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.242103    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-dshm\") pod \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\" (UID: \"0326dcc8-4daf-4971-bea8-7fb756f5b7fa\") "
Sep 16 18:46:04.242347 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.242188    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "0326dcc8-4daf-4971-bea8-7fb756f5b7fa" (UID: "0326dcc8-4daf-4971-bea8-7fb756f5b7fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:04.242347 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.242293    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-home" (OuterVolumeSpecName: "home") pod "0326dcc8-4daf-4971-bea8-7fb756f5b7fa" (UID: "0326dcc8-4daf-4971-bea8-7fb756f5b7fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:04.242511 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.242451    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-model-cache\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:46:04.242511 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.242467    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-home\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:46:04.242753 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.242710    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0326dcc8-4daf-4971-bea8-7fb756f5b7fa" (UID: "0326dcc8-4daf-4971-bea8-7fb756f5b7fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:04.244595 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.244565    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0326dcc8-4daf-4971-bea8-7fb756f5b7fa" (UID: "0326dcc8-4daf-4971-bea8-7fb756f5b7fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:04.244810 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.244780    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-dshm" (OuterVolumeSpecName: "dshm") pod "0326dcc8-4daf-4971-bea8-7fb756f5b7fa" (UID: "0326dcc8-4daf-4971-bea8-7fb756f5b7fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:04.244810 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.244793    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-kube-api-access-5fhcv" (OuterVolumeSpecName: "kube-api-access-5fhcv") pod "0326dcc8-4daf-4971-bea8-7fb756f5b7fa" (UID: "0326dcc8-4daf-4971-bea8-7fb756f5b7fa"). InnerVolumeSpecName "kube-api-access-5fhcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:04.343596 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.343555    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-tmp-dir\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:46:04.343596 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.343592    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-tls-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:46:04.343821 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.343610    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fhcv\" (UniqueName: \"kubernetes.io/projected/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-kube-api-access-5fhcv\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:46:04.343821 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.343623    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0326dcc8-4daf-4971-bea8-7fb756f5b7fa-dshm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:46:04.361882 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.361849    2554 generic.go:356] "Generic (PLEG): container finished" podID="0326dcc8-4daf-4971-bea8-7fb756f5b7fa" containerID="0c6a0e00c117947bcfbaf6e04ab1526866803e038df723962eb6b2edeb745a22" exitCode=0
Sep 16 18:46:04.362056 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.361905    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr" event={"ID":"0326dcc8-4daf-4971-bea8-7fb756f5b7fa","Type":"ContainerDied","Data":"0c6a0e00c117947bcfbaf6e04ab1526866803e038df723962eb6b2edeb745a22"}
Sep 16 18:46:04.362056 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.361935    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr" event={"ID":"0326dcc8-4daf-4971-bea8-7fb756f5b7fa","Type":"ContainerDied","Data":"90231f83b197d45a094ee4f33e178977b8296987c4a42923f397b42423ec36ba"}
Sep 16 18:46:04.362056 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.361952    2554 scope.go:117] "RemoveContainer" containerID="0c6a0e00c117947bcfbaf6e04ab1526866803e038df723962eb6b2edeb745a22"
Sep 16 18:46:04.362056 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.361906    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"
Sep 16 18:46:04.370957 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.370938    2554 scope.go:117] "RemoveContainer" containerID="0c6a0e00c117947bcfbaf6e04ab1526866803e038df723962eb6b2edeb745a22"
Sep 16 18:46:04.371220 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:46:04.371199    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c6a0e00c117947bcfbaf6e04ab1526866803e038df723962eb6b2edeb745a22\": container with ID starting with 0c6a0e00c117947bcfbaf6e04ab1526866803e038df723962eb6b2edeb745a22 not found: ID does not exist" containerID="0c6a0e00c117947bcfbaf6e04ab1526866803e038df723962eb6b2edeb745a22"
Sep 16 18:46:04.371320 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.371249    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6a0e00c117947bcfbaf6e04ab1526866803e038df723962eb6b2edeb745a22"} err="failed to get container status \"0c6a0e00c117947bcfbaf6e04ab1526866803e038df723962eb6b2edeb745a22\": rpc error: code = NotFound desc = could not find container \"0c6a0e00c117947bcfbaf6e04ab1526866803e038df723962eb6b2edeb745a22\": container with ID starting with 0c6a0e00c117947bcfbaf6e04ab1526866803e038df723962eb6b2edeb745a22 not found: ID does not exist"
Sep 16 18:46:04.383155 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.383132    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"]
Sep 16 18:46:04.385587 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.385568    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/auth-test-model-972df0-kserve-69b5858959-hdlmr"]
Sep 16 18:46:04.883468 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:04.883425    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0326dcc8-4daf-4971-bea8-7fb756f5b7fa" path="/var/lib/kubelet/pods/0326dcc8-4daf-4971-bea8-7fb756f5b7fa/volumes"
Sep 16 18:46:09.684056 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:09.683977    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:46:09.684499 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:09.684268    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:46:09.696251 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:09.696216    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:46:10.403287 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:10.403255    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:46:16.926593 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:16.926562    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:46:16.934118 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:16.934092    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:46:19.318221 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.318182    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"]
Sep 16 18:46:19.318718 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.318588    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0326dcc8-4daf-4971-bea8-7fb756f5b7fa" containerName="main"
Sep 16 18:46:19.318718 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.318600    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0326dcc8-4daf-4971-bea8-7fb756f5b7fa" containerName="main"
Sep 16 18:46:19.318718 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.318666    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0326dcc8-4daf-4971-bea8-7fb756f5b7fa" containerName="main"
Sep 16 18:46:19.321626 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.321606    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.325132 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.325105    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d3e8be\"/\"openshift-service-ca.crt\""
Sep 16 18:46:19.325945 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.325926    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d3e8be\"/\"default-dockercfg-wzmqh\""
Sep 16 18:46:19.328372 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.328352    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d3e8be\"/\"sub-test-model-d3e8be-kserve-self-signed-certs\""
Sep 16 18:46:19.328455 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.328428    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d3e8be\"/\"kube-root-ca.crt\""
Sep 16 18:46:19.343358 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.343334    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"]
Sep 16 18:46:19.486254 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.486196    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-home\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.486428 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.486262    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b681ecd6-6659-4118-a4f8-810f3537cc6f-tls-certs\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.486428 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.486295    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-dshm\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.486428 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.486341    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-tmp-dir\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.486428 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.486395    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-model-cache\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.486566 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.486454    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xzrw\" (UniqueName: \"kubernetes.io/projected/b681ecd6-6659-4118-a4f8-810f3537cc6f-kube-api-access-6xzrw\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.587673 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.587586    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b681ecd6-6659-4118-a4f8-810f3537cc6f-tls-certs\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.587673 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.587628    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-dshm\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.587673 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.587666    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-tmp-dir\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.587939 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.587708    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-model-cache\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.587939 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.587742    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xzrw\" (UniqueName: \"kubernetes.io/projected/b681ecd6-6659-4118-a4f8-810f3537cc6f-kube-api-access-6xzrw\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.587939 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.587767    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-home\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.588161 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.588136    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-tmp-dir\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.588161 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.588154    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-home\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.588290 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.588187    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-model-cache\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.589968 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.589948    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-dshm\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.590212 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.590195    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b681ecd6-6659-4118-a4f8-810f3537cc6f-tls-certs\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.596039 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.596019    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xzrw\" (UniqueName: \"kubernetes.io/projected/b681ecd6-6659-4118-a4f8-810f3537cc6f-kube-api-access-6xzrw\") pod \"sub-test-model-d3e8be-kserve-58c98f894f-v89zq\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.630589 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.630539    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:19.906335 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:19.906250    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"]
Sep 16 18:46:19.908919 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:46:19.908894    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb681ecd6_6659_4118_a4f8_810f3537cc6f.slice/crio-8ab255d8e4384b986bbc528f0e64d84a1bacb11e7cb938bfd0425016d54b59ff WatchSource:0}: Error finding container 8ab255d8e4384b986bbc528f0e64d84a1bacb11e7cb938bfd0425016d54b59ff: Status 404 returned error can't find the container with id 8ab255d8e4384b986bbc528f0e64d84a1bacb11e7cb938bfd0425016d54b59ff
Sep 16 18:46:20.451217 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:20.451169    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq" event={"ID":"b681ecd6-6659-4118-a4f8-810f3537cc6f","Type":"ContainerStarted","Data":"713c04e6cf30ca64b675e8866bf6631be37c47c1085150f9eedfa32dd50980eb"}
Sep 16 18:46:20.451217 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:20.451216    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq" event={"ID":"b681ecd6-6659-4118-a4f8-810f3537cc6f","Type":"ContainerStarted","Data":"8ab255d8e4384b986bbc528f0e64d84a1bacb11e7cb938bfd0425016d54b59ff"}
Sep 16 18:46:20.471526 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:20.471482    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq" podStartSLOduration=1.471467738 podStartE2EDuration="1.471467738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:20.469776652 +0000 UTC m=+1804.028944487" watchObservedRunningTime="2026-09-16 18:46:20.471467738 +0000 UTC m=+1804.030635572"
Sep 16 18:46:23.126591 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:23.126554    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:46:29.631130 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:29.631081    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:29.631602 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:29.631158    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:29.645103 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:29.645067    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:30.520150 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:30.520121    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:46:37.302649 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:37.302616    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:46:50.212463 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:50.212425    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:46:57.394095 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:57.394051    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"]
Sep 16 18:46:58.477130 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:46:58.477093    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"]
Sep 16 18:47:00.313610 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:00.313573    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:47:02.119162 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.119127    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"]
Sep 16 18:47:02.123090 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.123074    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.125323 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.125302    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-972df0\"/\"rate-test-model-972df0-kserve-self-signed-certs\""
Sep 16 18:47:02.132057 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.132037    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"]
Sep 16 18:47:02.286555 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.286507    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddb7w\" (UniqueName: \"kubernetes.io/projected/08663488-96e5-47d0-a1d7-9c28b74be6ff-kube-api-access-ddb7w\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.286743 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.286561    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08663488-96e5-47d0-a1d7-9c28b74be6ff-tls-certs\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.286743 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.286620    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-model-cache\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.286743 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.286696    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-dshm\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.286878 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.286758    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-home\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.286878 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.286789    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-tmp-dir\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.388071 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.387973    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-home\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.388071 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.388032    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-tmp-dir\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.388368 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.388109    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddb7w\" (UniqueName: \"kubernetes.io/projected/08663488-96e5-47d0-a1d7-9c28b74be6ff-kube-api-access-ddb7w\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.388368 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.388136    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08663488-96e5-47d0-a1d7-9c28b74be6ff-tls-certs\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.388368 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.388172    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-model-cache\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.388368 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.388316    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-dshm\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.388586 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.388447    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-home\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.388586 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.388564    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-tmp-dir\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.388670 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.388584    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-model-cache\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.390532 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.390509    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-dshm\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.390656 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.390639    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08663488-96e5-47d0-a1d7-9c28b74be6ff-tls-certs\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.396769 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.396747    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddb7w\" (UniqueName: \"kubernetes.io/projected/08663488-96e5-47d0-a1d7-9c28b74be6ff-kube-api-access-ddb7w\") pod \"rate-test-model-972df0-kserve-78649cb547-7lmd7\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") " pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.432251 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.432202    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:02.712243 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:02.712148    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"]
Sep 16 18:47:02.715151 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:47:02.715117    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08663488_96e5_47d0_a1d7_9c28b74be6ff.slice/crio-26cb8f6dbe2aedbced649943bff537ecc4a61b9f14829bb59710fc6b7b70f4c6 WatchSource:0}: Error finding container 26cb8f6dbe2aedbced649943bff537ecc4a61b9f14829bb59710fc6b7b70f4c6: Status 404 returned error can't find the container with id 26cb8f6dbe2aedbced649943bff537ecc4a61b9f14829bb59710fc6b7b70f4c6
Sep 16 18:47:03.684295 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:03.684259    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7" event={"ID":"08663488-96e5-47d0-a1d7-9c28b74be6ff","Type":"ContainerStarted","Data":"6c089da2848d699a3a60f5dd9a832443440fd30e95b51129f6029697411eae61"}
Sep 16 18:47:03.684766 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:03.684307    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7" event={"ID":"08663488-96e5-47d0-a1d7-9c28b74be6ff","Type":"ContainerStarted","Data":"26cb8f6dbe2aedbced649943bff537ecc4a61b9f14829bb59710fc6b7b70f4c6"}
Sep 16 18:47:03.702772 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:03.702716    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7" podStartSLOduration=1.702694235 podStartE2EDuration="1.702694235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:03.702520993 +0000 UTC m=+1847.261688894" watchObservedRunningTime="2026-09-16 18:47:03.702694235 +0000 UTC m=+1847.261862068"
Sep 16 18:47:11.019444 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:11.019412    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:47:12.412137 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.412096    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94" podUID="fdd89922-74c9-4749-a1c2-4ea00408957f" containerName="main" containerID="cri-o://beb0960bbd705b1211de4939eca1fd4596e84102217013c2fe03281e4191e72f" gracePeriod=45
Sep 16 18:47:12.432607 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.432582    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:12.432710 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.432658    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:12.445429 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.445401    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:12.657903 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.657875    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:47:12.681674 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.681593    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-tmp-dir\") pod \"fdd89922-74c9-4749-a1c2-4ea00408957f\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") "
Sep 16 18:47:12.681674 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.681664    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-dshm\") pod \"fdd89922-74c9-4749-a1c2-4ea00408957f\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") "
Sep 16 18:47:12.681884 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.681727    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m92dj\" (UniqueName: \"kubernetes.io/projected/fdd89922-74c9-4749-a1c2-4ea00408957f-kube-api-access-m92dj\") pod \"fdd89922-74c9-4749-a1c2-4ea00408957f\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") "
Sep 16 18:47:12.681884 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.681755    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-home\") pod \"fdd89922-74c9-4749-a1c2-4ea00408957f\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") "
Sep 16 18:47:12.681884 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.681795    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd89922-74c9-4749-a1c2-4ea00408957f-tls-certs\") pod \"fdd89922-74c9-4749-a1c2-4ea00408957f\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") "
Sep 16 18:47:12.681884 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.681861    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-model-cache\") pod \"fdd89922-74c9-4749-a1c2-4ea00408957f\" (UID: \"fdd89922-74c9-4749-a1c2-4ea00408957f\") "
Sep 16 18:47:12.682076 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.681910    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdd89922-74c9-4749-a1c2-4ea00408957f" (UID: "fdd89922-74c9-4749-a1c2-4ea00408957f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:12.682122 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.682064    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-home" (OuterVolumeSpecName: "home") pod "fdd89922-74c9-4749-a1c2-4ea00408957f" (UID: "fdd89922-74c9-4749-a1c2-4ea00408957f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:12.682190 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.682163    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-home\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:47:12.682190 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.682176    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-tmp-dir\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:47:12.682306 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.682284    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdd89922-74c9-4749-a1c2-4ea00408957f" (UID: "fdd89922-74c9-4749-a1c2-4ea00408957f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:12.683931 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.683905    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-dshm" (OuterVolumeSpecName: "dshm") pod "fdd89922-74c9-4749-a1c2-4ea00408957f" (UID: "fdd89922-74c9-4749-a1c2-4ea00408957f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:12.683931 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.683904    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdd89922-74c9-4749-a1c2-4ea00408957f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdd89922-74c9-4749-a1c2-4ea00408957f" (UID: "fdd89922-74c9-4749-a1c2-4ea00408957f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:12.684086 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.683928    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd89922-74c9-4749-a1c2-4ea00408957f-kube-api-access-m92dj" (OuterVolumeSpecName: "kube-api-access-m92dj") pod "fdd89922-74c9-4749-a1c2-4ea00408957f" (UID: "fdd89922-74c9-4749-a1c2-4ea00408957f"). InnerVolumeSpecName "kube-api-access-m92dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:12.730694 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.730660    2554 generic.go:356] "Generic (PLEG): container finished" podID="fdd89922-74c9-4749-a1c2-4ea00408957f" containerID="beb0960bbd705b1211de4939eca1fd4596e84102217013c2fe03281e4191e72f" exitCode=0
Sep 16 18:47:12.730832 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.730701    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"
Sep 16 18:47:12.730832 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.730743    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94" event={"ID":"fdd89922-74c9-4749-a1c2-4ea00408957f","Type":"ContainerDied","Data":"beb0960bbd705b1211de4939eca1fd4596e84102217013c2fe03281e4191e72f"}
Sep 16 18:47:12.730832 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.730786    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94" event={"ID":"fdd89922-74c9-4749-a1c2-4ea00408957f","Type":"ContainerDied","Data":"005c99788a9975c762825636537c78d1c8eb809701c59a256ca2a7e362e5142b"}
Sep 16 18:47:12.730832 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.730803    2554 scope.go:117] "RemoveContainer" containerID="beb0960bbd705b1211de4939eca1fd4596e84102217013c2fe03281e4191e72f"
Sep 16 18:47:12.739727 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.739708    2554 scope.go:117] "RemoveContainer" containerID="beb0960bbd705b1211de4939eca1fd4596e84102217013c2fe03281e4191e72f"
Sep 16 18:47:12.739981 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:47:12.739961    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb0960bbd705b1211de4939eca1fd4596e84102217013c2fe03281e4191e72f\": container with ID starting with beb0960bbd705b1211de4939eca1fd4596e84102217013c2fe03281e4191e72f not found: ID does not exist" containerID="beb0960bbd705b1211de4939eca1fd4596e84102217013c2fe03281e4191e72f"
Sep 16 18:47:12.740056 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.739993    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb0960bbd705b1211de4939eca1fd4596e84102217013c2fe03281e4191e72f"} err="failed to get container status \"beb0960bbd705b1211de4939eca1fd4596e84102217013c2fe03281e4191e72f\": rpc error: code = NotFound desc = could not find container \"beb0960bbd705b1211de4939eca1fd4596e84102217013c2fe03281e4191e72f\": container with ID starting with beb0960bbd705b1211de4939eca1fd4596e84102217013c2fe03281e4191e72f not found: ID does not exist"
Sep 16 18:47:12.743393 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.743373    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:47:12.753603 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.753582    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"]
Sep 16 18:47:12.759585 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.759556    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/sub-test-model-972df0-kserve-59f8b55475-lqc94"]
Sep 16 18:47:12.783114 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.783088    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd89922-74c9-4749-a1c2-4ea00408957f-tls-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:47:12.783114 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.783112    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-model-cache\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:47:12.783272 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.783120    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd89922-74c9-4749-a1c2-4ea00408957f-dshm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:47:12.783272 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.783132    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m92dj\" (UniqueName: \"kubernetes.io/projected/fdd89922-74c9-4749-a1c2-4ea00408957f-kube-api-access-m92dj\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:47:12.889528 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:12.886202    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdd89922-74c9-4749-a1c2-4ea00408957f" path="/var/lib/kubelet/pods/fdd89922-74c9-4749-a1c2-4ea00408957f/volumes"
Sep 16 18:47:13.496181 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.496131    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq" podUID="b681ecd6-6659-4118-a4f8-810f3537cc6f" containerName="main" containerID="cri-o://713c04e6cf30ca64b675e8866bf6631be37c47c1085150f9eedfa32dd50980eb" gracePeriod=45
Sep 16 18:47:13.738699 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.738670    2554 generic.go:356] "Generic (PLEG): container finished" podID="b681ecd6-6659-4118-a4f8-810f3537cc6f" containerID="713c04e6cf30ca64b675e8866bf6631be37c47c1085150f9eedfa32dd50980eb" exitCode=0
Sep 16 18:47:13.738816 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.738743    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq" event={"ID":"b681ecd6-6659-4118-a4f8-810f3537cc6f","Type":"ContainerDied","Data":"713c04e6cf30ca64b675e8866bf6631be37c47c1085150f9eedfa32dd50980eb"}
Sep 16 18:47:13.738816 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.738780    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq" event={"ID":"b681ecd6-6659-4118-a4f8-810f3537cc6f","Type":"ContainerDied","Data":"8ab255d8e4384b986bbc528f0e64d84a1bacb11e7cb938bfd0425016d54b59ff"}
Sep 16 18:47:13.738816 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.738793    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ab255d8e4384b986bbc528f0e64d84a1bacb11e7cb938bfd0425016d54b59ff"
Sep 16 18:47:13.756072 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.756001    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:47:13.792050 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.792023    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-model-cache\") pod \"b681ecd6-6659-4118-a4f8-810f3537cc6f\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") "
Sep 16 18:47:13.792219 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.792104    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-home\") pod \"b681ecd6-6659-4118-a4f8-810f3537cc6f\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") "
Sep 16 18:47:13.792219 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.792170    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b681ecd6-6659-4118-a4f8-810f3537cc6f-tls-certs\") pod \"b681ecd6-6659-4118-a4f8-810f3537cc6f\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") "
Sep 16 18:47:13.792386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.792227    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xzrw\" (UniqueName: \"kubernetes.io/projected/b681ecd6-6659-4118-a4f8-810f3537cc6f-kube-api-access-6xzrw\") pod \"b681ecd6-6659-4118-a4f8-810f3537cc6f\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") "
Sep 16 18:47:13.792386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.792275    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-dshm\") pod \"b681ecd6-6659-4118-a4f8-810f3537cc6f\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") "
Sep 16 18:47:13.792386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.792275    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-home" (OuterVolumeSpecName: "home") pod "b681ecd6-6659-4118-a4f8-810f3537cc6f" (UID: "b681ecd6-6659-4118-a4f8-810f3537cc6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:13.792386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.792316    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b681ecd6-6659-4118-a4f8-810f3537cc6f" (UID: "b681ecd6-6659-4118-a4f8-810f3537cc6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:13.792386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.792337    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-tmp-dir\") pod \"b681ecd6-6659-4118-a4f8-810f3537cc6f\" (UID: \"b681ecd6-6659-4118-a4f8-810f3537cc6f\") "
Sep 16 18:47:13.792657 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.792639    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b681ecd6-6659-4118-a4f8-810f3537cc6f" (UID: "b681ecd6-6659-4118-a4f8-810f3537cc6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:13.792719 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.792647    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-model-cache\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:47:13.792719 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.792675    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-home\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:47:13.794491 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.794463    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b681ecd6-6659-4118-a4f8-810f3537cc6f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b681ecd6-6659-4118-a4f8-810f3537cc6f" (UID: "b681ecd6-6659-4118-a4f8-810f3537cc6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:13.794491 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.794483    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-dshm" (OuterVolumeSpecName: "dshm") pod "b681ecd6-6659-4118-a4f8-810f3537cc6f" (UID: "b681ecd6-6659-4118-a4f8-810f3537cc6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:13.794741 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.794718    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b681ecd6-6659-4118-a4f8-810f3537cc6f-kube-api-access-6xzrw" (OuterVolumeSpecName: "kube-api-access-6xzrw") pod "b681ecd6-6659-4118-a4f8-810f3537cc6f" (UID: "b681ecd6-6659-4118-a4f8-810f3537cc6f"). InnerVolumeSpecName "kube-api-access-6xzrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:13.893164 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.893128    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xzrw\" (UniqueName: \"kubernetes.io/projected/b681ecd6-6659-4118-a4f8-810f3537cc6f-kube-api-access-6xzrw\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:47:13.893164 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.893159    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-dshm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:47:13.893164 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.893169    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b681ecd6-6659-4118-a4f8-810f3537cc6f-tmp-dir\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:47:13.893417 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:13.893176    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b681ecd6-6659-4118-a4f8-810f3537cc6f-tls-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:47:14.743500 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:14.743465    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"
Sep 16 18:47:14.769084 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:14.769046    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"]
Sep 16 18:47:14.770847 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:14.770825    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d3e8be/sub-test-model-d3e8be-kserve-58c98f894f-v89zq"]
Sep 16 18:47:14.883674 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:14.883641    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b681ecd6-6659-4118-a4f8-810f3537cc6f" path="/var/lib/kubelet/pods/b681ecd6-6659-4118-a4f8-810f3537cc6f/volumes"
Sep 16 18:47:39.418528 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.418435    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"]
Sep 16 18:47:39.419027 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.418838    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd89922-74c9-4749-a1c2-4ea00408957f" containerName="main"
Sep 16 18:47:39.419027 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.418849    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd89922-74c9-4749-a1c2-4ea00408957f" containerName="main"
Sep 16 18:47:39.419027 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.418862    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b681ecd6-6659-4118-a4f8-810f3537cc6f" containerName="main"
Sep 16 18:47:39.419027 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.418868    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b681ecd6-6659-4118-a4f8-810f3537cc6f" containerName="main"
Sep 16 18:47:39.419027 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.418932    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b681ecd6-6659-4118-a4f8-810f3537cc6f" containerName="main"
Sep 16 18:47:39.419027 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.418941    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd89922-74c9-4749-a1c2-4ea00408957f" containerName="main"
Sep 16 18:47:39.422082 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.422053    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.424509 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.424490    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d3e8be\"/\"openshift-service-ca.crt\""
Sep 16 18:47:39.424643 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.424528    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d3e8be\"/\"default-dockercfg-wzmqh\""
Sep 16 18:47:39.425403 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.425381    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d3e8be\"/\"kube-root-ca.crt\""
Sep 16 18:47:39.425487 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.425462    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d3e8be\"/\"rate-test-model-d3e8be-kserve-self-signed-certs\""
Sep 16 18:47:39.437702 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.437678    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"]
Sep 16 18:47:39.527659 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.527620    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515b6c18-05fc-428b-be69-a845034b93ed-tls-certs\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.527659 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.527657    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-dshm\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.527875 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.527683    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-model-cache\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.527875 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.527750    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-home\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.527875 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.527777    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-tmp-dir\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.527875 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.527809    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dc7c\" (UniqueName: \"kubernetes.io/projected/515b6c18-05fc-428b-be69-a845034b93ed-kube-api-access-9dc7c\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.628332 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.628281    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515b6c18-05fc-428b-be69-a845034b93ed-tls-certs\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.628332 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.628326    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-dshm\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.628615 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.628353    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-model-cache\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.628615 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.628401    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-home\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.628615 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.628431    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-tmp-dir\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.628615 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.628469    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dc7c\" (UniqueName: \"kubernetes.io/projected/515b6c18-05fc-428b-be69-a845034b93ed-kube-api-access-9dc7c\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.628862 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.628834    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-home\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.628915 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.628866    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-model-cache\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.628952 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.628932    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-tmp-dir\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.630631 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.630608    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-dshm\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.631014 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.630996    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515b6c18-05fc-428b-be69-a845034b93ed-tls-certs\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.636873 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.636848    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dc7c\" (UniqueName: \"kubernetes.io/projected/515b6c18-05fc-428b-be69-a845034b93ed-kube-api-access-9dc7c\") pod \"rate-test-model-d3e8be-kserve-5b99448d7f-fnchj\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") " pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:39.731872 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:39.731834    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:40.011070 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:40.011042    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"]
Sep 16 18:47:40.012906 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:47:40.012876    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod515b6c18_05fc_428b_be69_a845034b93ed.slice/crio-3c7ce75a368c17602ee539af97b88cc47d15eb1c1491ae499d04fe2f7e8b42fd WatchSource:0}: Error finding container 3c7ce75a368c17602ee539af97b88cc47d15eb1c1491ae499d04fe2f7e8b42fd: Status 404 returned error can't find the container with id 3c7ce75a368c17602ee539af97b88cc47d15eb1c1491ae499d04fe2f7e8b42fd
Sep 16 18:47:40.875616 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:40.875584    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj" event={"ID":"515b6c18-05fc-428b-be69-a845034b93ed","Type":"ContainerStarted","Data":"56343bdd40626db23cecb65dd7e6d22876b6a19ca6352e38769827b0f70a9cec"}
Sep 16 18:47:40.876013 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:40.875630    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj" event={"ID":"515b6c18-05fc-428b-be69-a845034b93ed","Type":"ContainerStarted","Data":"3c7ce75a368c17602ee539af97b88cc47d15eb1c1491ae499d04fe2f7e8b42fd"}
Sep 16 18:47:40.902013 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:40.901963    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj" podStartSLOduration=1.901946827 podStartE2EDuration="1.901946827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:40.899150606 +0000 UTC m=+1884.458318440" watchObservedRunningTime="2026-09-16 18:47:40.901946827 +0000 UTC m=+1884.461114658"
Sep 16 18:47:43.126955 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:43.126920    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:47:49.732566 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:49.732528    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:49.733166 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:49.732839    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:49.744731 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:49.744695    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:47:49.928991 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:47:49.928961    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:48:06.420798 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:06.420748    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:48:15.515211 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:15.515171    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:48:22.481098 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:22.481057    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"]
Sep 16 18:48:23.307217 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:23.307180    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"]
Sep 16 18:48:24.063454 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.063419    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"]
Sep 16 18:48:24.067032 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.067011    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.069618 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.069593    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:48:24.069734 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.069599    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:48:24.069880 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.069863    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6817d3-openshift-default-dockercfg-9l679\""
Sep 16 18:48:24.071438 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.071420    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:48:24.089617 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.089595    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"]
Sep 16 18:48:24.240791 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.240742    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-envoy\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.240984 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.240805    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2c5r\" (UniqueName: \"kubernetes.io/projected/987af540-96fe-4ac8-844c-3aa916cf50dc-kube-api-access-k2c5r\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.240984 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.240868    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/987af540-96fe-4ac8-844c-3aa916cf50dc-istiod-ca-cert\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.240984 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.240893    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-data\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.240984 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.240917    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-credential-socket\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.240984 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.240947    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-workload-socket\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.241266 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.241028    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-token\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.241266 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.241063    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-podinfo\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.241266 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.241098    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-workload-certs\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.342682 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.342592    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-token\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.342682 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.342633    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-podinfo\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.342682 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.342667    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-workload-certs\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.342952 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.342694    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-envoy\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.342952 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.342718    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2c5r\" (UniqueName: \"kubernetes.io/projected/987af540-96fe-4ac8-844c-3aa916cf50dc-kube-api-access-k2c5r\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.342952 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.342768    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/987af540-96fe-4ac8-844c-3aa916cf50dc-istiod-ca-cert\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.342952 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.342797    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-data\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.342952 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.342836    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-credential-socket\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.342952 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.342876    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-workload-socket\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.343455 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.343199    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-workload-certs\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.343530 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.343455    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-credential-socket\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.343703 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.343677    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/987af540-96fe-4ac8-844c-3aa916cf50dc-istiod-ca-cert\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.343703 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.343697    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-workload-socket\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.343885 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.343869    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-data\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.345130 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.345104    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-envoy\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.345581 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.345560    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-podinfo\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.352007 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.351986    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2c5r\" (UniqueName: \"kubernetes.io/projected/987af540-96fe-4ac8-844c-3aa916cf50dc-kube-api-access-k2c5r\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.352594 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.352573    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-token\") pod \"e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") " pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.376652 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.376614    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:24.653982 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.653957    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"]
Sep 16 18:48:24.656077 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:48:24.656042    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987af540_96fe_4ac8_844c_3aa916cf50dc.slice/crio-55b6f5c510577284001cee42807a4dc5aef4bef4d19f97a5e43f288c55b4ae09 WatchSource:0}: Error finding container 55b6f5c510577284001cee42807a4dc5aef4bef4d19f97a5e43f288c55b4ae09: Status 404 returned error can't find the container with id 55b6f5c510577284001cee42807a4dc5aef4bef4d19f97a5e43f288c55b4ae09
Sep 16 18:48:24.658486 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.658448    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:48:24.658574 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.658528    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:48:24.658574 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:24.658556    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:48:25.108881 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:25.108845    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv" event={"ID":"987af540-96fe-4ac8-844c-3aa916cf50dc","Type":"ContainerStarted","Data":"31e5efa17837df162c29c0183b387ffd0b98e5a5f42f3aa340da6b7c748466b9"}
Sep 16 18:48:25.109349 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:25.108897    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv" event={"ID":"987af540-96fe-4ac8-844c-3aa916cf50dc","Type":"ContainerStarted","Data":"55b6f5c510577284001cee42807a4dc5aef4bef4d19f97a5e43f288c55b4ae09"}
Sep 16 18:48:25.134146 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:25.134100    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv" podStartSLOduration=1.1340844429999999 podStartE2EDuration="1.134084443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:25.132643045 +0000 UTC m=+1928.691810865" watchObservedRunningTime="2026-09-16 18:48:25.134084443 +0000 UTC m=+1928.693252276"
Sep 16 18:48:25.377724 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:25.377646    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:25.382205 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:25.382179    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:25.816270 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:25.816240    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:48:26.114443 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:26.114277    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:26.129519 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:26.115222    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:30.353893 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.353862    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"]
Sep 16 18:48:30.357315 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.357298    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:48:30.360036 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.360020    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6817d3-serving-cert\""
Sep 16 18:48:30.376409 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.376388    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"]
Sep 16 18:48:30.497449 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.497417    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66bjm\" (UniqueName: \"kubernetes.io/projected/2170aee3-5439-489c-a94b-c5b46663666c-kube-api-access-66bjm\") pod \"maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm\" (UID: \"2170aee3-5439-489c-a94b-c5b46663666c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:48:30.497607 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.497470    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2170aee3-5439-489c-a94b-c5b46663666c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm\" (UID: \"2170aee3-5439-489c-a94b-c5b46663666c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:48:30.497656 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.497613    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2170aee3-5439-489c-a94b-c5b46663666c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm\" (UID: \"2170aee3-5439-489c-a94b-c5b46663666c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:48:30.497693 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.497658    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2170aee3-5439-489c-a94b-c5b46663666c-metrics-certs\") pod \"maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm\" (UID: \"2170aee3-5439-489c-a94b-c5b46663666c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:48:30.598487 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.598463    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2170aee3-5439-489c-a94b-c5b46663666c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm\" (UID: \"2170aee3-5439-489c-a94b-c5b46663666c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:48:30.598607 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.598496    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2170aee3-5439-489c-a94b-c5b46663666c-metrics-certs\") pod \"maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm\" (UID: \"2170aee3-5439-489c-a94b-c5b46663666c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:48:30.598607 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.598526    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66bjm\" (UniqueName: \"kubernetes.io/projected/2170aee3-5439-489c-a94b-c5b46663666c-kube-api-access-66bjm\") pod \"maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm\" (UID: \"2170aee3-5439-489c-a94b-c5b46663666c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:48:30.598607 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.598556    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2170aee3-5439-489c-a94b-c5b46663666c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm\" (UID: \"2170aee3-5439-489c-a94b-c5b46663666c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:48:30.599133 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.599108    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2170aee3-5439-489c-a94b-c5b46663666c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm\" (UID: \"2170aee3-5439-489c-a94b-c5b46663666c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:48:30.600913 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.600891    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2170aee3-5439-489c-a94b-c5b46663666c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm\" (UID: \"2170aee3-5439-489c-a94b-c5b46663666c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:48:30.601017 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.600933    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2170aee3-5439-489c-a94b-c5b46663666c-metrics-certs\") pod \"maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm\" (UID: \"2170aee3-5439-489c-a94b-c5b46663666c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:48:30.613608 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.613560    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66bjm\" (UniqueName: \"kubernetes.io/projected/2170aee3-5439-489c-a94b-c5b46663666c-kube-api-access-66bjm\") pod \"maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm\" (UID: \"2170aee3-5439-489c-a94b-c5b46663666c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:48:30.667557 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.667537    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:48:30.942694 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.942655    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"]
Sep 16 18:48:30.945082 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:48:30.945054    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2170aee3_5439_489c_a94b_c5b46663666c.slice/crio-3fa717ccfc365e811f0146bc8fffc5814de6fb4e023ebe09a9162c44590f8afc WatchSource:0}: Error finding container 3fa717ccfc365e811f0146bc8fffc5814de6fb4e023ebe09a9162c44590f8afc: Status 404 returned error can't find the container with id 3fa717ccfc365e811f0146bc8fffc5814de6fb4e023ebe09a9162c44590f8afc
Sep 16 18:48:30.946892 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:30.946869    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:48:31.139791 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.139753    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm" event={"ID":"2170aee3-5439-489c-a94b-c5b46663666c","Type":"ContainerStarted","Data":"3fa717ccfc365e811f0146bc8fffc5814de6fb4e023ebe09a9162c44590f8afc"}
Sep 16 18:48:31.320758 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.320721    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"]
Sep 16 18:48:31.325430 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.325404    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"
Sep 16 18:48:31.327921 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.327902    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6817d3\""
Sep 16 18:48:31.328141 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.328125    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6817d3-dockercfg-gc9hh\""
Sep 16 18:48:31.335730 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.335706    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"]
Sep 16 18:48:31.421429 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.421393    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"]
Sep 16 18:48:31.425484 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.425462    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"
Sep 16 18:48:31.444499 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.444050    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"]
Sep 16 18:48:31.506470 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.506445    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703190ae-ec9d-44f3-893c-d67518deed1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc\" (UID: \"703190ae-ec9d-44f3-893c-d67518deed1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"
Sep 16 18:48:31.506578 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.506565    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsq29\" (UniqueName: \"kubernetes.io/projected/703190ae-ec9d-44f3-893c-d67518deed1c-kube-api-access-qsq29\") pod \"payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc\" (UID: \"703190ae-ec9d-44f3-893c-d67518deed1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"
Sep 16 18:48:31.607950 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.607873    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703190ae-ec9d-44f3-893c-d67518deed1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc\" (UID: \"703190ae-ec9d-44f3-893c-d67518deed1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"
Sep 16 18:48:31.608110 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.607965    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqwnr\" (UniqueName: \"kubernetes.io/projected/55270654-9067-49d3-8b75-f249da99d76b-kube-api-access-vqwnr\") pod \"payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5\" (UID: \"55270654-9067-49d3-8b75-f249da99d76b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"
Sep 16 18:48:31.608110 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.607997    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55270654-9067-49d3-8b75-f249da99d76b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5\" (UID: \"55270654-9067-49d3-8b75-f249da99d76b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"
Sep 16 18:48:31.608110 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.608065    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsq29\" (UniqueName: \"kubernetes.io/projected/703190ae-ec9d-44f3-893c-d67518deed1c-kube-api-access-qsq29\") pod \"payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc\" (UID: \"703190ae-ec9d-44f3-893c-d67518deed1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"
Sep 16 18:48:31.608659 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.608640    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703190ae-ec9d-44f3-893c-d67518deed1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc\" (UID: \"703190ae-ec9d-44f3-893c-d67518deed1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"
Sep 16 18:48:31.619332 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.619303    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsq29\" (UniqueName: \"kubernetes.io/projected/703190ae-ec9d-44f3-893c-d67518deed1c-kube-api-access-qsq29\") pod \"payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc\" (UID: \"703190ae-ec9d-44f3-893c-d67518deed1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"
Sep 16 18:48:31.636726 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.636704    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"
Sep 16 18:48:31.709601 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.709568    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqwnr\" (UniqueName: \"kubernetes.io/projected/55270654-9067-49d3-8b75-f249da99d76b-kube-api-access-vqwnr\") pod \"payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5\" (UID: \"55270654-9067-49d3-8b75-f249da99d76b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"
Sep 16 18:48:31.709764 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.709616    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55270654-9067-49d3-8b75-f249da99d76b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5\" (UID: \"55270654-9067-49d3-8b75-f249da99d76b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"
Sep 16 18:48:31.711158 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.711104    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55270654-9067-49d3-8b75-f249da99d76b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5\" (UID: \"55270654-9067-49d3-8b75-f249da99d76b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"
Sep 16 18:48:31.718418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.718397    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqwnr\" (UniqueName: \"kubernetes.io/projected/55270654-9067-49d3-8b75-f249da99d76b-kube-api-access-vqwnr\") pod \"payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5\" (UID: \"55270654-9067-49d3-8b75-f249da99d76b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"
Sep 16 18:48:31.735363 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.735338    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"
Sep 16 18:48:31.921750 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:31.921727    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"]
Sep 16 18:48:31.922958 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:48:31.922932    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703190ae_ec9d_44f3_893c_d67518deed1c.slice/crio-2b8c59c106d6457a57f6e33b45d590ed1ce918b9705561b0354dfd9eddd6de18 WatchSource:0}: Error finding container 2b8c59c106d6457a57f6e33b45d590ed1ce918b9705561b0354dfd9eddd6de18: Status 404 returned error can't find the container with id 2b8c59c106d6457a57f6e33b45d590ed1ce918b9705561b0354dfd9eddd6de18
Sep 16 18:48:32.029646 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:32.029615    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"]
Sep 16 18:48:32.032769 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:48:32.032738    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55270654_9067_49d3_8b75_f249da99d76b.slice/crio-e79560ac7ab00549221954a263320862e78d4e09314aff4e0050f82e556a2b34 WatchSource:0}: Error finding container e79560ac7ab00549221954a263320862e78d4e09314aff4e0050f82e556a2b34: Status 404 returned error can't find the container with id e79560ac7ab00549221954a263320862e78d4e09314aff4e0050f82e556a2b34
Sep 16 18:48:32.149308 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:32.149250    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm" event={"ID":"2170aee3-5439-489c-a94b-c5b46663666c","Type":"ContainerStarted","Data":"f4235decbfab1521bb34dec79b22b4ab134fa3e45f6035521c89423524418db7"}
Sep 16 18:48:32.149481 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:32.149322    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:48:32.151300 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:32.151270    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5" event={"ID":"55270654-9067-49d3-8b75-f249da99d76b","Type":"ContainerStarted","Data":"0f11eed3e1227393793ace3270e3e9b0c42efcca186fc60b781d59afa351a1cf"}
Sep 16 18:48:32.151439 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:32.151327    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"
Sep 16 18:48:32.151439 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:32.151352    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5" event={"ID":"55270654-9067-49d3-8b75-f249da99d76b","Type":"ContainerStarted","Data":"e79560ac7ab00549221954a263320862e78d4e09314aff4e0050f82e556a2b34"}
Sep 16 18:48:32.152799 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:32.152781    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc" event={"ID":"703190ae-ec9d-44f3-893c-d67518deed1c","Type":"ContainerStarted","Data":"092aa125b09d502305bd16ce67e9c254f1f7dd149d6140488c0571fdc0ee93b2"}
Sep 16 18:48:32.152950 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:32.152818    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc" event={"ID":"703190ae-ec9d-44f3-893c-d67518deed1c","Type":"ContainerStarted","Data":"2b8c59c106d6457a57f6e33b45d590ed1ce918b9705561b0354dfd9eddd6de18"}
Sep 16 18:48:32.152950 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:32.152846    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"
Sep 16 18:48:32.172602 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:32.172519    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm" podStartSLOduration=1.615551044 podStartE2EDuration="2.172502115s" totalImagesPullingTime="556.951071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:32.170367791 +0000 UTC m=+1935.729535621" watchObservedRunningTime="2026-09-16 18:48:32.172502115 +0000 UTC m=+1935.731669946"
Sep 16 18:48:32.191424 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:32.191382    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5" podStartSLOduration=1.191366754 podStartE2EDuration="1.191366754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:32.189186497 +0000 UTC m=+1935.748354330" watchObservedRunningTime="2026-09-16 18:48:32.191366754 +0000 UTC m=+1935.750534586"
Sep 16 18:48:32.206505 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:32.206454    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc" podStartSLOduration=1.206428349 podStartE2EDuration="1.206428349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:32.204517224 +0000 UTC m=+1935.763685051" watchObservedRunningTime="2026-09-16 18:48:32.206428349 +0000 UTC m=+1935.765596174"
Sep 16 18:48:35.821125 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:35.821087    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:48:37.499639 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.499595    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7" podUID="08663488-96e5-47d0-a1d7-9c28b74be6ff" containerName="main" containerID="cri-o://6c089da2848d699a3a60f5dd9a832443440fd30e95b51129f6029697411eae61" gracePeriod=45
Sep 16 18:48:37.795793 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.795770    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:48:37.871724 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.871686    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-dshm\") pod \"08663488-96e5-47d0-a1d7-9c28b74be6ff\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") "
Sep 16 18:48:37.871724 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.871727    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddb7w\" (UniqueName: \"kubernetes.io/projected/08663488-96e5-47d0-a1d7-9c28b74be6ff-kube-api-access-ddb7w\") pod \"08663488-96e5-47d0-a1d7-9c28b74be6ff\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") "
Sep 16 18:48:37.871921 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.871783    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08663488-96e5-47d0-a1d7-9c28b74be6ff-tls-certs\") pod \"08663488-96e5-47d0-a1d7-9c28b74be6ff\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") "
Sep 16 18:48:37.871921 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.871850    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-model-cache\") pod \"08663488-96e5-47d0-a1d7-9c28b74be6ff\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") "
Sep 16 18:48:37.871921 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.871879    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-home\") pod \"08663488-96e5-47d0-a1d7-9c28b74be6ff\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") "
Sep 16 18:48:37.871921 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.871902    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-tmp-dir\") pod \"08663488-96e5-47d0-a1d7-9c28b74be6ff\" (UID: \"08663488-96e5-47d0-a1d7-9c28b74be6ff\") "
Sep 16 18:48:37.872126 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.872085    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "08663488-96e5-47d0-a1d7-9c28b74be6ff" (UID: "08663488-96e5-47d0-a1d7-9c28b74be6ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:37.872126 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.872106    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-home" (OuterVolumeSpecName: "home") pod "08663488-96e5-47d0-a1d7-9c28b74be6ff" (UID: "08663488-96e5-47d0-a1d7-9c28b74be6ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:37.872226 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.872173    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08663488-96e5-47d0-a1d7-9c28b74be6ff" (UID: "08663488-96e5-47d0-a1d7-9c28b74be6ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:37.872308 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.872272    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-model-cache\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:48:37.872308 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.872286    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-home\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:48:37.872308 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.872294    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-tmp-dir\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:48:37.873984 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.873957    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-dshm" (OuterVolumeSpecName: "dshm") pod "08663488-96e5-47d0-a1d7-9c28b74be6ff" (UID: "08663488-96e5-47d0-a1d7-9c28b74be6ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:37.873984 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.873964    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08663488-96e5-47d0-a1d7-9c28b74be6ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08663488-96e5-47d0-a1d7-9c28b74be6ff" (UID: "08663488-96e5-47d0-a1d7-9c28b74be6ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:37.873984 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.873976    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08663488-96e5-47d0-a1d7-9c28b74be6ff-kube-api-access-ddb7w" (OuterVolumeSpecName: "kube-api-access-ddb7w") pod "08663488-96e5-47d0-a1d7-9c28b74be6ff" (UID: "08663488-96e5-47d0-a1d7-9c28b74be6ff"). InnerVolumeSpecName "kube-api-access-ddb7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:37.973035 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.972998    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08663488-96e5-47d0-a1d7-9c28b74be6ff-dshm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:48:37.973035 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.973028    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddb7w\" (UniqueName: \"kubernetes.io/projected/08663488-96e5-47d0-a1d7-9c28b74be6ff-kube-api-access-ddb7w\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:48:37.973035 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:37.973039    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08663488-96e5-47d0-a1d7-9c28b74be6ff-tls-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:48:38.176598 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.172341    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:48:38.189340 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.189311    2554 generic.go:356] "Generic (PLEG): container finished" podID="08663488-96e5-47d0-a1d7-9c28b74be6ff" containerID="6c089da2848d699a3a60f5dd9a832443440fd30e95b51129f6029697411eae61" exitCode=0
Sep 16 18:48:38.189462 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.189356    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"
Sep 16 18:48:38.189462 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.189373    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7" event={"ID":"08663488-96e5-47d0-a1d7-9c28b74be6ff","Type":"ContainerDied","Data":"6c089da2848d699a3a60f5dd9a832443440fd30e95b51129f6029697411eae61"}
Sep 16 18:48:38.189462 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.189402    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7" event={"ID":"08663488-96e5-47d0-a1d7-9c28b74be6ff","Type":"ContainerDied","Data":"26cb8f6dbe2aedbced649943bff537ecc4a61b9f14829bb59710fc6b7b70f4c6"}
Sep 16 18:48:38.189462 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.189417    2554 scope.go:117] "RemoveContainer" containerID="6c089da2848d699a3a60f5dd9a832443440fd30e95b51129f6029697411eae61"
Sep 16 18:48:38.199758 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.199737    2554 scope.go:117] "RemoveContainer" containerID="6c089da2848d699a3a60f5dd9a832443440fd30e95b51129f6029697411eae61"
Sep 16 18:48:38.200024 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:48:38.200005    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c089da2848d699a3a60f5dd9a832443440fd30e95b51129f6029697411eae61\": container with ID starting with 6c089da2848d699a3a60f5dd9a832443440fd30e95b51129f6029697411eae61 not found: ID does not exist" containerID="6c089da2848d699a3a60f5dd9a832443440fd30e95b51129f6029697411eae61"
Sep 16 18:48:38.200082 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.200033    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c089da2848d699a3a60f5dd9a832443440fd30e95b51129f6029697411eae61"} err="failed to get container status \"6c089da2848d699a3a60f5dd9a832443440fd30e95b51129f6029697411eae61\": rpc error: code = NotFound desc = could not find container \"6c089da2848d699a3a60f5dd9a832443440fd30e95b51129f6029697411eae61\": container with ID starting with 6c089da2848d699a3a60f5dd9a832443440fd30e95b51129f6029697411eae61 not found: ID does not exist"
Sep 16 18:48:38.221525 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.221504    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"]
Sep 16 18:48:38.232157 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.232135    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-972df0/rate-test-model-972df0-kserve-78649cb547-7lmd7"]
Sep 16 18:48:38.325630 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.325596    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj" podUID="515b6c18-05fc-428b-be69-a845034b93ed" containerName="main" containerID="cri-o://56343bdd40626db23cecb65dd7e6d22876b6a19ca6352e38769827b0f70a9cec" gracePeriod=45
Sep 16 18:48:38.620801 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.620778    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:48:38.681255 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.681204    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-dshm\") pod \"515b6c18-05fc-428b-be69-a845034b93ed\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") "
Sep 16 18:48:38.681403 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.681262    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-home\") pod \"515b6c18-05fc-428b-be69-a845034b93ed\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") "
Sep 16 18:48:38.681403 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.681290    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-tmp-dir\") pod \"515b6c18-05fc-428b-be69-a845034b93ed\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") "
Sep 16 18:48:38.681403 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.681337    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-model-cache\") pod \"515b6c18-05fc-428b-be69-a845034b93ed\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") "
Sep 16 18:48:38.681403 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.681363    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dc7c\" (UniqueName: \"kubernetes.io/projected/515b6c18-05fc-428b-be69-a845034b93ed-kube-api-access-9dc7c\") pod \"515b6c18-05fc-428b-be69-a845034b93ed\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") "
Sep 16 18:48:38.681621 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.681421    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515b6c18-05fc-428b-be69-a845034b93ed-tls-certs\") pod \"515b6c18-05fc-428b-be69-a845034b93ed\" (UID: \"515b6c18-05fc-428b-be69-a845034b93ed\") "
Sep 16 18:48:38.681621 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.681512    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "515b6c18-05fc-428b-be69-a845034b93ed" (UID: "515b6c18-05fc-428b-be69-a845034b93ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:38.681621 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.681527    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-home" (OuterVolumeSpecName: "home") pod "515b6c18-05fc-428b-be69-a845034b93ed" (UID: "515b6c18-05fc-428b-be69-a845034b93ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:38.681621 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.681593    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "515b6c18-05fc-428b-be69-a845034b93ed" (UID: "515b6c18-05fc-428b-be69-a845034b93ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:38.681819 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.681743    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-home\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:48:38.681819 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.681762    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-tmp-dir\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:48:38.681819 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.681774    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-model-cache\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:48:38.683364 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.683337    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515b6c18-05fc-428b-be69-a845034b93ed-kube-api-access-9dc7c" (OuterVolumeSpecName: "kube-api-access-9dc7c") pod "515b6c18-05fc-428b-be69-a845034b93ed" (UID: "515b6c18-05fc-428b-be69-a845034b93ed"). InnerVolumeSpecName "kube-api-access-9dc7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:38.683691 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.683677    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-dshm" (OuterVolumeSpecName: "dshm") pod "515b6c18-05fc-428b-be69-a845034b93ed" (UID: "515b6c18-05fc-428b-be69-a845034b93ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:38.683757 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.683721    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/515b6c18-05fc-428b-be69-a845034b93ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "515b6c18-05fc-428b-be69-a845034b93ed" (UID: "515b6c18-05fc-428b-be69-a845034b93ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:38.782633 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.782597    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dc7c\" (UniqueName: \"kubernetes.io/projected/515b6c18-05fc-428b-be69-a845034b93ed-kube-api-access-9dc7c\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:48:38.782633 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.782624    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515b6c18-05fc-428b-be69-a845034b93ed-tls-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:48:38.782633 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.782636    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515b6c18-05fc-428b-be69-a845034b93ed-dshm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:48:38.885324 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:38.885289    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08663488-96e5-47d0-a1d7-9c28b74be6ff" path="/var/lib/kubelet/pods/08663488-96e5-47d0-a1d7-9c28b74be6ff/volumes"
Sep 16 18:48:39.197474 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:39.197437    2554 generic.go:356] "Generic (PLEG): container finished" podID="515b6c18-05fc-428b-be69-a845034b93ed" containerID="56343bdd40626db23cecb65dd7e6d22876b6a19ca6352e38769827b0f70a9cec" exitCode=0
Sep 16 18:48:39.197474 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:39.197474    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"
Sep 16 18:48:39.197731 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:39.197515    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj" event={"ID":"515b6c18-05fc-428b-be69-a845034b93ed","Type":"ContainerDied","Data":"56343bdd40626db23cecb65dd7e6d22876b6a19ca6352e38769827b0f70a9cec"}
Sep 16 18:48:39.197731 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:39.197558    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj" event={"ID":"515b6c18-05fc-428b-be69-a845034b93ed","Type":"ContainerDied","Data":"3c7ce75a368c17602ee539af97b88cc47d15eb1c1491ae499d04fe2f7e8b42fd"}
Sep 16 18:48:39.197731 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:39.197581    2554 scope.go:117] "RemoveContainer" containerID="56343bdd40626db23cecb65dd7e6d22876b6a19ca6352e38769827b0f70a9cec"
Sep 16 18:48:39.206191 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:39.206173    2554 scope.go:117] "RemoveContainer" containerID="56343bdd40626db23cecb65dd7e6d22876b6a19ca6352e38769827b0f70a9cec"
Sep 16 18:48:39.206465 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:48:39.206449    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56343bdd40626db23cecb65dd7e6d22876b6a19ca6352e38769827b0f70a9cec\": container with ID starting with 56343bdd40626db23cecb65dd7e6d22876b6a19ca6352e38769827b0f70a9cec not found: ID does not exist" containerID="56343bdd40626db23cecb65dd7e6d22876b6a19ca6352e38769827b0f70a9cec"
Sep 16 18:48:39.206539 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:39.206473    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56343bdd40626db23cecb65dd7e6d22876b6a19ca6352e38769827b0f70a9cec"} err="failed to get container status \"56343bdd40626db23cecb65dd7e6d22876b6a19ca6352e38769827b0f70a9cec\": rpc error: code = NotFound desc = could not find container \"56343bdd40626db23cecb65dd7e6d22876b6a19ca6352e38769827b0f70a9cec\": container with ID starting with 56343bdd40626db23cecb65dd7e6d22876b6a19ca6352e38769827b0f70a9cec not found: ID does not exist"
Sep 16 18:48:39.218669 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:39.218644    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"]
Sep 16 18:48:39.223555 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:39.223536    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d3e8be/rate-test-model-d3e8be-kserve-5b99448d7f-fnchj"]
Sep 16 18:48:40.883781 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:40.883748    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="515b6c18-05fc-428b-be69-a845034b93ed" path="/var/lib/kubelet/pods/515b6c18-05fc-428b-be69-a845034b93ed/volumes"
Sep 16 18:48:43.162322 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:43.162285    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"
Sep 16 18:48:43.162709 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:43.162381    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"
Sep 16 18:48:45.237146 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:45.237116    2554 generic.go:356] "Generic (PLEG): container finished" podID="987af540-96fe-4ac8-844c-3aa916cf50dc" containerID="31e5efa17837df162c29c0183b387ffd0b98e5a5f42f3aa340da6b7c748466b9" exitCode=0
Sep 16 18:48:45.237660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:45.237186    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv" event={"ID":"987af540-96fe-4ac8-844c-3aa916cf50dc","Type":"ContainerDied","Data":"31e5efa17837df162c29c0183b387ffd0b98e5a5f42f3aa340da6b7c748466b9"}
Sep 16 18:48:45.237725 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:45.237685    2554 scope.go:117] "RemoveContainer" containerID="31e5efa17837df162c29c0183b387ffd0b98e5a5f42f3aa340da6b7c748466b9"
Sep 16 18:48:45.238761 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:45.238734    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:48:45.238830 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:45.238801    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:48:45.238865 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:45.238832    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:48:46.247217 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:46.247174    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv" event={"ID":"987af540-96fe-4ac8-844c-3aa916cf50dc","Type":"ContainerStarted","Data":"a8d807436980b40044176302736326635dd04bb27329ef6a25ffe8086e2f19c1"}
Sep 16 18:48:46.376768 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:46.376726    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:46.377091 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:46.377066    2554 patch_prober.go:28] interesting pod/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 16 18:48:46.377189 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:46.377127    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv" podUID="987af540-96fe-4ac8-844c-3aa916cf50dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 16 18:48:47.377500 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:47.377463    2554 patch_prober.go:28] interesting pod/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 16 18:48:47.377876 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:47.377512    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv" podUID="987af540-96fe-4ac8-844c-3aa916cf50dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 16 18:48:48.377348 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:48.377292    2554 patch_prober.go:28] interesting pod/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 16 18:48:48.377554 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:48.377457    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv" podUID="987af540-96fe-4ac8-844c-3aa916cf50dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 16 18:48:49.377765 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:49.377725    2554 patch_prober.go:28] interesting pod/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 16 18:48:49.378159 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:49.377801    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv" podUID="987af540-96fe-4ac8-844c-3aa916cf50dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 16 18:48:50.377193 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:50.377155    2554 patch_prober.go:28] interesting pod/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 16 18:48:50.377405 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:50.377260    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv" podUID="987af540-96fe-4ac8-844c-3aa916cf50dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 16 18:48:51.380940 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:51.380901    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:51.381394 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:51.381303    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:48:51.381949 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:48:51.381931    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:49:04.318461 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.318425    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"]
Sep 16 18:49:04.319296 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.318828    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="515b6c18-05fc-428b-be69-a845034b93ed" containerName="main"
Sep 16 18:49:04.319296 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.318838    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="515b6c18-05fc-428b-be69-a845034b93ed" containerName="main"
Sep 16 18:49:04.319296 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.318849    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08663488-96e5-47d0-a1d7-9c28b74be6ff" containerName="main"
Sep 16 18:49:04.319296 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.318854    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="08663488-96e5-47d0-a1d7-9c28b74be6ff" containerName="main"
Sep 16 18:49:04.319296 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.318922    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="515b6c18-05fc-428b-be69-a845034b93ed" containerName="main"
Sep 16 18:49:04.319296 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.318933    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="08663488-96e5-47d0-a1d7-9c28b74be6ff" containerName="main"
Sep 16 18:49:04.320949 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.320930    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:04.323466 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.323440    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c90bd2-serving-cert\""
Sep 16 18:49:04.336846 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.336822    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"]
Sep 16 18:49:04.426394 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.426359    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4213288-7ddc-40d7-bfa2-4b929bef4703-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:04.426568 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.426443    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4213288-7ddc-40d7-bfa2-4b929bef4703-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:04.426568 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.426488    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4213288-7ddc-40d7-bfa2-4b929bef4703-metrics-certs\") pod \"maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:04.426568 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.426507    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqz87\" (UniqueName: \"kubernetes.io/projected/c4213288-7ddc-40d7-bfa2-4b929bef4703-kube-api-access-sqz87\") pod \"maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:04.527738 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.527702    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4213288-7ddc-40d7-bfa2-4b929bef4703-metrics-certs\") pod \"maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:04.527738 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.527740    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqz87\" (UniqueName: \"kubernetes.io/projected/c4213288-7ddc-40d7-bfa2-4b929bef4703-kube-api-access-sqz87\") pod \"maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:04.527982 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.527765    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4213288-7ddc-40d7-bfa2-4b929bef4703-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:04.527982 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.527852    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4213288-7ddc-40d7-bfa2-4b929bef4703-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:04.527982 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:49:04.527944    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-serving-cert: secret "maas-api-e2e-ipp-b-c90bd2-serving-cert" not found
Sep 16 18:49:04.528139 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:49:04.528025    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4213288-7ddc-40d7-bfa2-4b929bef4703-maas-api-tls podName:c4213288-7ddc-40d7-bfa2-4b929bef4703 nodeName:}" failed. No retries permitted until 2026-09-16 18:49:05.028001978 +0000 UTC m=+1968.587169790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4213288-7ddc-40d7-bfa2-4b929bef4703-maas-api-tls") pod "maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9" (UID: "c4213288-7ddc-40d7-bfa2-4b929bef4703") : secret "maas-api-e2e-ipp-b-c90bd2-serving-cert" not found
Sep 16 18:49:04.528487 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.528469    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4213288-7ddc-40d7-bfa2-4b929bef4703-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:04.530309 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.530289    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4213288-7ddc-40d7-bfa2-4b929bef4703-metrics-certs\") pod \"maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:04.536873 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.536852    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqz87\" (UniqueName: \"kubernetes.io/projected/c4213288-7ddc-40d7-bfa2-4b929bef4703-kube-api-access-sqz87\") pod \"maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:04.957573 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.957535    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"]
Sep 16 18:49:04.960616 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.960599    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"
Sep 16 18:49:04.962762 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.962739    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c90bd2\""
Sep 16 18:49:04.975099 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.975074    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"]
Sep 16 18:49:04.993301 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.993253    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"]
Sep 16 18:49:04.995602 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.995587    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"
Sep 16 18:49:04.997803 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:04.997785    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c90bd2-dockercfg-h5h49\""
Sep 16 18:49:05.007152 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.007130    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"]
Sep 16 18:49:05.032900 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.032863    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bef44bf0-e6e4-43af-801b-3ea0a13e061b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz\" (UID: \"bef44bf0-e6e4-43af-801b-3ea0a13e061b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"
Sep 16 18:49:05.033033 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.032964    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4213288-7ddc-40d7-bfa2-4b929bef4703-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:05.033033 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.033007    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfl44\" (UniqueName: \"kubernetes.io/projected/bef44bf0-e6e4-43af-801b-3ea0a13e061b-kube-api-access-rfl44\") pod \"payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz\" (UID: \"bef44bf0-e6e4-43af-801b-3ea0a13e061b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"
Sep 16 18:49:05.035099 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.035079    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4213288-7ddc-40d7-bfa2-4b929bef4703-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:05.134192 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.134154    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfl44\" (UniqueName: \"kubernetes.io/projected/bef44bf0-e6e4-43af-801b-3ea0a13e061b-kube-api-access-rfl44\") pod \"payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz\" (UID: \"bef44bf0-e6e4-43af-801b-3ea0a13e061b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"
Sep 16 18:49:05.134412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.134218    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c73dd77-1850-4d0e-81df-0602437534fb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4\" (UID: \"6c73dd77-1850-4d0e-81df-0602437534fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"
Sep 16 18:49:05.134412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.134272    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsp6h\" (UniqueName: \"kubernetes.io/projected/6c73dd77-1850-4d0e-81df-0602437534fb-kube-api-access-lsp6h\") pod \"payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4\" (UID: \"6c73dd77-1850-4d0e-81df-0602437534fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"
Sep 16 18:49:05.134412 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.134360    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bef44bf0-e6e4-43af-801b-3ea0a13e061b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz\" (UID: \"bef44bf0-e6e4-43af-801b-3ea0a13e061b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"
Sep 16 18:49:05.135050 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.135031    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bef44bf0-e6e4-43af-801b-3ea0a13e061b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz\" (UID: \"bef44bf0-e6e4-43af-801b-3ea0a13e061b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"
Sep 16 18:49:05.144248 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.144199    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfl44\" (UniqueName: \"kubernetes.io/projected/bef44bf0-e6e4-43af-801b-3ea0a13e061b-kube-api-access-rfl44\") pod \"payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz\" (UID: \"bef44bf0-e6e4-43af-801b-3ea0a13e061b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"
Sep 16 18:49:05.232132 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.232098    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:05.235288 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.235268    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c73dd77-1850-4d0e-81df-0602437534fb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4\" (UID: \"6c73dd77-1850-4d0e-81df-0602437534fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"
Sep 16 18:49:05.235377 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.235301    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsp6h\" (UniqueName: \"kubernetes.io/projected/6c73dd77-1850-4d0e-81df-0602437534fb-kube-api-access-lsp6h\") pod \"payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4\" (UID: \"6c73dd77-1850-4d0e-81df-0602437534fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"
Sep 16 18:49:05.235933 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.235908    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c73dd77-1850-4d0e-81df-0602437534fb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4\" (UID: \"6c73dd77-1850-4d0e-81df-0602437534fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"
Sep 16 18:49:05.243824 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.243797    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsp6h\" (UniqueName: \"kubernetes.io/projected/6c73dd77-1850-4d0e-81df-0602437534fb-kube-api-access-lsp6h\") pod \"payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4\" (UID: \"6c73dd77-1850-4d0e-81df-0602437534fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"
Sep 16 18:49:05.271282 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.271222    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"
Sep 16 18:49:05.305842 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.305807    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"
Sep 16 18:49:05.507346 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.507319    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"]
Sep 16 18:49:05.508827 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:49:05.508803    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4213288_7ddc_40d7_bfa2_4b929bef4703.slice/crio-7c72498088fff2f41035af42a81634d515f76a002f50b3e6cccca56d0f50dc98 WatchSource:0}: Error finding container 7c72498088fff2f41035af42a81634d515f76a002f50b3e6cccca56d0f50dc98: Status 404 returned error can't find the container with id 7c72498088fff2f41035af42a81634d515f76a002f50b3e6cccca56d0f50dc98
Sep 16 18:49:05.554623 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.554599    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"]
Sep 16 18:49:05.556324 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:49:05.556301    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef44bf0_e6e4_43af_801b_3ea0a13e061b.slice/crio-4e4a18b312e8c0f588d15bf1ebee02c44c725bd53daec3278bbd04f049e70da6 WatchSource:0}: Error finding container 4e4a18b312e8c0f588d15bf1ebee02c44c725bd53daec3278bbd04f049e70da6: Status 404 returned error can't find the container with id 4e4a18b312e8c0f588d15bf1ebee02c44c725bd53daec3278bbd04f049e70da6
Sep 16 18:49:05.592767 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:05.592741    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"]
Sep 16 18:49:05.595110 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:49:05.595080    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c73dd77_1850_4d0e_81df_0602437534fb.slice/crio-83f378265590f72f153af774b2fa5a0cd953d081f8f4c709e40887e837bc0b01 WatchSource:0}: Error finding container 83f378265590f72f153af774b2fa5a0cd953d081f8f4c709e40887e837bc0b01: Status 404 returned error can't find the container with id 83f378265590f72f153af774b2fa5a0cd953d081f8f4c709e40887e837bc0b01
Sep 16 18:49:06.362912 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:06.362870    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz" event={"ID":"bef44bf0-e6e4-43af-801b-3ea0a13e061b","Type":"ContainerStarted","Data":"2ab8536ecdb1a64ab127d62d3c319d509bc5f5a03bac02cbf1793d61cd72835e"}
Sep 16 18:49:06.363127 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:06.362959    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz" event={"ID":"bef44bf0-e6e4-43af-801b-3ea0a13e061b","Type":"ContainerStarted","Data":"4e4a18b312e8c0f588d15bf1ebee02c44c725bd53daec3278bbd04f049e70da6"}
Sep 16 18:49:06.363127 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:06.363054    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"
Sep 16 18:49:06.364468 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:06.364445    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9" event={"ID":"c4213288-7ddc-40d7-bfa2-4b929bef4703","Type":"ContainerStarted","Data":"1722f5387a0429c94737c01f429f998e96defc8e6f784657b0d2c7947265ab83"}
Sep 16 18:49:06.364586 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:06.364497    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:06.364586 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:06.364523    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9" event={"ID":"c4213288-7ddc-40d7-bfa2-4b929bef4703","Type":"ContainerStarted","Data":"7c72498088fff2f41035af42a81634d515f76a002f50b3e6cccca56d0f50dc98"}
Sep 16 18:49:06.365994 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:06.365972    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4" event={"ID":"6c73dd77-1850-4d0e-81df-0602437534fb","Type":"ContainerStarted","Data":"523420269b2367080cee12efc1f6ef89283c5d36bd903aeba8b621015905528c"}
Sep 16 18:49:06.366102 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:06.366007    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4" event={"ID":"6c73dd77-1850-4d0e-81df-0602437534fb","Type":"ContainerStarted","Data":"83f378265590f72f153af774b2fa5a0cd953d081f8f4c709e40887e837bc0b01"}
Sep 16 18:49:06.366167 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:06.366110    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"
Sep 16 18:49:06.380405 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:06.380360    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz" podStartSLOduration=2.380347614 podStartE2EDuration="2.380347614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:06.378947558 +0000 UTC m=+1969.938115392" watchObservedRunningTime="2026-09-16 18:49:06.380347614 +0000 UTC m=+1969.939515447"
Sep 16 18:49:06.398817 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:06.398772    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9" podStartSLOduration=2.05973542 podStartE2EDuration="2.398757184s" totalImagesPullingTime="339.021764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:06.396766096 +0000 UTC m=+1969.955933929" watchObservedRunningTime="2026-09-16 18:49:06.398757184 +0000 UTC m=+1969.957925016"
Sep 16 18:49:06.414134 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:06.414095    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4" podStartSLOduration=2.4140810679999998 podStartE2EDuration="2.414081068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:06.412965517 +0000 UTC m=+1969.972133349" watchObservedRunningTime="2026-09-16 18:49:06.414081068 +0000 UTC m=+1969.973248900"
Sep 16 18:49:09.122804 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:09.122717    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:49:12.376982 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:12.376953    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:49:17.374082 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:17.374049    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"
Sep 16 18:49:17.374536 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:17.374211    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"
Sep 16 18:49:31.942246 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:31.942213    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4_6c73dd77-1850-4d0e-81df-0602437534fb/payload-processing/0.log"
Sep 16 18:49:50.855386 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:49:50.855345    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:50:11.617567 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:50:11.617526    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:50:17.996464 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:50:17.996414    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5_55270654-9067-49d3-8b75-f249da99d76b/payload-processing/0.log"
Sep 16 18:50:18.222900 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:50:18.222870    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4_6c73dd77-1850-4d0e-81df-0602437534fb/payload-processing/0.log"
Sep 16 18:50:30.906254 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:50:30.906202    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:51:08.106497 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:08.106463    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:51:16.960380 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:16.960351    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:51:16.971835 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:16.971810    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:51:27.504380 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:27.504340    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:51:29.929998 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:29.929960    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"]
Sep 16 18:51:29.930487 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:29.930255    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm" podUID="2170aee3-5439-489c-a94b-c5b46663666c" containerName="maas-api" containerID="cri-o://f4235decbfab1521bb34dec79b22b4ab134fa3e45f6035521c89423524418db7" gracePeriod=30
Sep 16 18:51:30.004066 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.004033    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"]
Sep 16 18:51:30.004476 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.004296    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5" podUID="55270654-9067-49d3-8b75-f249da99d76b" containerName="payload-processing" containerID="cri-o://0f11eed3e1227393793ace3270e3e9b0c42efcca186fc60b781d59afa351a1cf" gracePeriod=30
Sep 16 18:51:30.012873 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.012813    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"]
Sep 16 18:51:30.013069 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.013042    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc" podUID="703190ae-ec9d-44f3-893c-d67518deed1c" containerName="payload-pre-processing" containerID="cri-o://092aa125b09d502305bd16ce67e9c254f1f7dd149d6140488c0571fdc0ee93b2" gracePeriod=30
Sep 16 18:51:30.138729 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.138691    2554 generic.go:356] "Generic (PLEG): container finished" podID="2170aee3-5439-489c-a94b-c5b46663666c" containerID="f4235decbfab1521bb34dec79b22b4ab134fa3e45f6035521c89423524418db7" exitCode=0
Sep 16 18:51:30.138873 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.138765    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm" event={"ID":"2170aee3-5439-489c-a94b-c5b46663666c","Type":"ContainerDied","Data":"f4235decbfab1521bb34dec79b22b4ab134fa3e45f6035521c89423524418db7"}
Sep 16 18:51:30.140898 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.140731    2554 generic.go:356] "Generic (PLEG): container finished" podID="55270654-9067-49d3-8b75-f249da99d76b" containerID="0f11eed3e1227393793ace3270e3e9b0c42efcca186fc60b781d59afa351a1cf" exitCode=0
Sep 16 18:51:30.140898 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.140822    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5" event={"ID":"55270654-9067-49d3-8b75-f249da99d76b","Type":"ContainerDied","Data":"0f11eed3e1227393793ace3270e3e9b0c42efcca186fc60b781d59afa351a1cf"}
Sep 16 18:51:30.143291 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.143119    2554 generic.go:356] "Generic (PLEG): container finished" podID="703190ae-ec9d-44f3-893c-d67518deed1c" containerID="092aa125b09d502305bd16ce67e9c254f1f7dd149d6140488c0571fdc0ee93b2" exitCode=0
Sep 16 18:51:30.143291 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.143204    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc" event={"ID":"703190ae-ec9d-44f3-893c-d67518deed1c","Type":"ContainerDied","Data":"092aa125b09d502305bd16ce67e9c254f1f7dd149d6140488c0571fdc0ee93b2"}
Sep 16 18:51:30.284196 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.284171    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:51:30.305665 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.305585    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"
Sep 16 18:51:30.337675 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.337648    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"
Sep 16 18:51:30.382362 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.382333    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2170aee3-5439-489c-a94b-c5b46663666c-metrics-certs\") pod \"2170aee3-5439-489c-a94b-c5b46663666c\" (UID: \"2170aee3-5439-489c-a94b-c5b46663666c\") "
Sep 16 18:51:30.382536 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.382378    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqwnr\" (UniqueName: \"kubernetes.io/projected/55270654-9067-49d3-8b75-f249da99d76b-kube-api-access-vqwnr\") pod \"55270654-9067-49d3-8b75-f249da99d76b\" (UID: \"55270654-9067-49d3-8b75-f249da99d76b\") "
Sep 16 18:51:30.382536 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.382412    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55270654-9067-49d3-8b75-f249da99d76b-plugins-config-volume\") pod \"55270654-9067-49d3-8b75-f249da99d76b\" (UID: \"55270654-9067-49d3-8b75-f249da99d76b\") "
Sep 16 18:51:30.382654 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.382557    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2170aee3-5439-489c-a94b-c5b46663666c-openshift-service-ca\") pod \"2170aee3-5439-489c-a94b-c5b46663666c\" (UID: \"2170aee3-5439-489c-a94b-c5b46663666c\") "
Sep 16 18:51:30.382714 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.382653    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2170aee3-5439-489c-a94b-c5b46663666c-maas-api-tls\") pod \"2170aee3-5439-489c-a94b-c5b46663666c\" (UID: \"2170aee3-5439-489c-a94b-c5b46663666c\") "
Sep 16 18:51:30.382714 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.382684    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66bjm\" (UniqueName: \"kubernetes.io/projected/2170aee3-5439-489c-a94b-c5b46663666c-kube-api-access-66bjm\") pod \"2170aee3-5439-489c-a94b-c5b46663666c\" (UID: \"2170aee3-5439-489c-a94b-c5b46663666c\") "
Sep 16 18:51:30.382818 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.382788    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55270654-9067-49d3-8b75-f249da99d76b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "55270654-9067-49d3-8b75-f249da99d76b" (UID: "55270654-9067-49d3-8b75-f249da99d76b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:51:30.383069 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.383050    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55270654-9067-49d3-8b75-f249da99d76b-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:51:30.383242 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.383198    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2170aee3-5439-489c-a94b-c5b46663666c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2170aee3-5439-489c-a94b-c5b46663666c" (UID: "2170aee3-5439-489c-a94b-c5b46663666c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:51:30.384623 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.384589    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2170aee3-5439-489c-a94b-c5b46663666c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2170aee3-5439-489c-a94b-c5b46663666c" (UID: "2170aee3-5439-489c-a94b-c5b46663666c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:51:30.384791 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.384769    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2170aee3-5439-489c-a94b-c5b46663666c-kube-api-access-66bjm" (OuterVolumeSpecName: "kube-api-access-66bjm") pod "2170aee3-5439-489c-a94b-c5b46663666c" (UID: "2170aee3-5439-489c-a94b-c5b46663666c"). InnerVolumeSpecName "kube-api-access-66bjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:30.384900 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.384880    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55270654-9067-49d3-8b75-f249da99d76b-kube-api-access-vqwnr" (OuterVolumeSpecName: "kube-api-access-vqwnr") pod "55270654-9067-49d3-8b75-f249da99d76b" (UID: "55270654-9067-49d3-8b75-f249da99d76b"). InnerVolumeSpecName "kube-api-access-vqwnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:30.385148 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.385133    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2170aee3-5439-489c-a94b-c5b46663666c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2170aee3-5439-489c-a94b-c5b46663666c" (UID: "2170aee3-5439-489c-a94b-c5b46663666c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:51:30.484465 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.484435    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703190ae-ec9d-44f3-893c-d67518deed1c-plugins-config-volume\") pod \"703190ae-ec9d-44f3-893c-d67518deed1c\" (UID: \"703190ae-ec9d-44f3-893c-d67518deed1c\") "
Sep 16 18:51:30.484655 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.484552    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsq29\" (UniqueName: \"kubernetes.io/projected/703190ae-ec9d-44f3-893c-d67518deed1c-kube-api-access-qsq29\") pod \"703190ae-ec9d-44f3-893c-d67518deed1c\" (UID: \"703190ae-ec9d-44f3-893c-d67518deed1c\") "
Sep 16 18:51:30.484829 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.484806    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/703190ae-ec9d-44f3-893c-d67518deed1c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "703190ae-ec9d-44f3-893c-d67518deed1c" (UID: "703190ae-ec9d-44f3-893c-d67518deed1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:51:30.484966 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.484952    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2170aee3-5439-489c-a94b-c5b46663666c-openshift-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:51:30.485008 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.484968    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2170aee3-5439-489c-a94b-c5b46663666c-maas-api-tls\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:51:30.485008 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.484977    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66bjm\" (UniqueName: \"kubernetes.io/projected/2170aee3-5439-489c-a94b-c5b46663666c-kube-api-access-66bjm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:51:30.485008 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.484985    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703190ae-ec9d-44f3-893c-d67518deed1c-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:51:30.485008 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.484994    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2170aee3-5439-489c-a94b-c5b46663666c-metrics-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:51:30.485008 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.485001    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqwnr\" (UniqueName: \"kubernetes.io/projected/55270654-9067-49d3-8b75-f249da99d76b-kube-api-access-vqwnr\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:51:30.486525 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.486496    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/703190ae-ec9d-44f3-893c-d67518deed1c-kube-api-access-qsq29" (OuterVolumeSpecName: "kube-api-access-qsq29") pod "703190ae-ec9d-44f3-893c-d67518deed1c" (UID: "703190ae-ec9d-44f3-893c-d67518deed1c"). InnerVolumeSpecName "kube-api-access-qsq29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:30.586489 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:30.586448    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsq29\" (UniqueName: \"kubernetes.io/projected/703190ae-ec9d-44f3-893c-d67518deed1c-kube-api-access-qsq29\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:51:31.151079 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:31.151000    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"
Sep 16 18:51:31.151079 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:31.151003    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5" event={"ID":"55270654-9067-49d3-8b75-f249da99d76b","Type":"ContainerDied","Data":"e79560ac7ab00549221954a263320862e78d4e09314aff4e0050f82e556a2b34"}
Sep 16 18:51:31.151610 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:31.151089    2554 scope.go:117] "RemoveContainer" containerID="0f11eed3e1227393793ace3270e3e9b0c42efcca186fc60b781d59afa351a1cf"
Sep 16 18:51:31.152740 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:31.152719    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"
Sep 16 18:51:31.152876 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:31.152756    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc" event={"ID":"703190ae-ec9d-44f3-893c-d67518deed1c","Type":"ContainerDied","Data":"2b8c59c106d6457a57f6e33b45d590ed1ce918b9705561b0354dfd9eddd6de18"}
Sep 16 18:51:31.154613 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:31.154593    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm" event={"ID":"2170aee3-5439-489c-a94b-c5b46663666c","Type":"ContainerDied","Data":"3fa717ccfc365e811f0146bc8fffc5814de6fb4e023ebe09a9162c44590f8afc"}
Sep 16 18:51:31.154714 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:31.154596    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"
Sep 16 18:51:31.160923 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:31.160902    2554 scope.go:117] "RemoveContainer" containerID="092aa125b09d502305bd16ce67e9c254f1f7dd149d6140488c0571fdc0ee93b2"
Sep 16 18:51:31.168747 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:31.168733    2554 scope.go:117] "RemoveContainer" containerID="f4235decbfab1521bb34dec79b22b4ab134fa3e45f6035521c89423524418db7"
Sep 16 18:51:31.175353 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:31.175331    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"]
Sep 16 18:51:31.179604 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:31.179585    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6817d3-8fc474599-qtbn5"]
Sep 16 18:51:31.191020 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:31.191001    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"]
Sep 16 18:51:31.194573 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:31.194552    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6817d3-6cf9ffc7f5-r49cm"]
Sep 16 18:51:31.205250 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:31.205222    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"]
Sep 16 18:51:31.215294 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:31.215274    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6817d3-8688849f4-xhpqc"]
Sep 16 18:51:32.884239 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:32.884193    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2170aee3-5439-489c-a94b-c5b46663666c" path="/var/lib/kubelet/pods/2170aee3-5439-489c-a94b-c5b46663666c/volumes"
Sep 16 18:51:32.884628 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:32.884586    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55270654-9067-49d3-8b75-f249da99d76b" path="/var/lib/kubelet/pods/55270654-9067-49d3-8b75-f249da99d76b/volumes"
Sep 16 18:51:32.884917 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:32.884904    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="703190ae-ec9d-44f3-893c-d67518deed1c" path="/var/lib/kubelet/pods/703190ae-ec9d-44f3-893c-d67518deed1c/volumes"
Sep 16 18:51:40.502210 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:51:40.502173    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:52:20.036819 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:20.036780    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"]
Sep 16 18:52:20.037397 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:20.037080    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv" podUID="987af540-96fe-4ac8-844c-3aa916cf50dc" containerName="istio-proxy" containerID="cri-o://a8d807436980b40044176302736326635dd04bb27329ef6a25ffe8086e2f19c1" gracePeriod=30
Sep 16 18:52:21.383687 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:21.383649    2554 patch_prober.go:28] interesting pod/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 18:52:21.384060 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:21.383710    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv" podUID="987af540-96fe-4ac8-844c-3aa916cf50dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:25.291687 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.291661    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:52:25.402660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.402569    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/987af540-96fe-4ac8-844c-3aa916cf50dc-istiod-ca-cert\") pod \"987af540-96fe-4ac8-844c-3aa916cf50dc\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") "
Sep 16 18:52:25.402660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.402611    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-podinfo\") pod \"987af540-96fe-4ac8-844c-3aa916cf50dc\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") "
Sep 16 18:52:25.402660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.402636    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-credential-socket\") pod \"987af540-96fe-4ac8-844c-3aa916cf50dc\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") "
Sep 16 18:52:25.402926 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.402667    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-token\") pod \"987af540-96fe-4ac8-844c-3aa916cf50dc\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") "
Sep 16 18:52:25.402926 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.402728    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-envoy\") pod \"987af540-96fe-4ac8-844c-3aa916cf50dc\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") "
Sep 16 18:52:25.402926 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.402765    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2c5r\" (UniqueName: \"kubernetes.io/projected/987af540-96fe-4ac8-844c-3aa916cf50dc-kube-api-access-k2c5r\") pod \"987af540-96fe-4ac8-844c-3aa916cf50dc\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") "
Sep 16 18:52:25.402926 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.402839    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-workload-socket\") pod \"987af540-96fe-4ac8-844c-3aa916cf50dc\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") "
Sep 16 18:52:25.402926 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.402877    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-workload-certs\") pod \"987af540-96fe-4ac8-844c-3aa916cf50dc\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") "
Sep 16 18:52:25.402926 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.402910    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-data\") pod \"987af540-96fe-4ac8-844c-3aa916cf50dc\" (UID: \"987af540-96fe-4ac8-844c-3aa916cf50dc\") "
Sep 16 18:52:25.403269 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.403000    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987af540-96fe-4ac8-844c-3aa916cf50dc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "987af540-96fe-4ac8-844c-3aa916cf50dc" (UID: "987af540-96fe-4ac8-844c-3aa916cf50dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:25.403269 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.403030    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "987af540-96fe-4ac8-844c-3aa916cf50dc" (UID: "987af540-96fe-4ac8-844c-3aa916cf50dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:25.403269 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.403257    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/987af540-96fe-4ac8-844c-3aa916cf50dc-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:25.403438 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.403279    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:25.403438 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.403296    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-data" (OuterVolumeSpecName: "istio-data") pod "987af540-96fe-4ac8-844c-3aa916cf50dc" (UID: "987af540-96fe-4ac8-844c-3aa916cf50dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:25.403438 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.403369    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "987af540-96fe-4ac8-844c-3aa916cf50dc" (UID: "987af540-96fe-4ac8-844c-3aa916cf50dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:25.403597 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.403463    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "987af540-96fe-4ac8-844c-3aa916cf50dc" (UID: "987af540-96fe-4ac8-844c-3aa916cf50dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:25.405015 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.404985    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "987af540-96fe-4ac8-844c-3aa916cf50dc" (UID: "987af540-96fe-4ac8-844c-3aa916cf50dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:52:25.405015 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.404987    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "987af540-96fe-4ac8-844c-3aa916cf50dc" (UID: "987af540-96fe-4ac8-844c-3aa916cf50dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:25.405338 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.405316    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-token" (OuterVolumeSpecName: "istio-token") pod "987af540-96fe-4ac8-844c-3aa916cf50dc" (UID: "987af540-96fe-4ac8-844c-3aa916cf50dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:25.405437 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.405349    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987af540-96fe-4ac8-844c-3aa916cf50dc-kube-api-access-k2c5r" (OuterVolumeSpecName: "kube-api-access-k2c5r") pod "987af540-96fe-4ac8-844c-3aa916cf50dc" (UID: "987af540-96fe-4ac8-844c-3aa916cf50dc"). InnerVolumeSpecName "kube-api-access-k2c5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:25.439317 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.439287    2554 generic.go:356] "Generic (PLEG): container finished" podID="987af540-96fe-4ac8-844c-3aa916cf50dc" containerID="a8d807436980b40044176302736326635dd04bb27329ef6a25ffe8086e2f19c1" exitCode=0
Sep 16 18:52:25.439451 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.439350    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"
Sep 16 18:52:25.439451 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.439374    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv" event={"ID":"987af540-96fe-4ac8-844c-3aa916cf50dc","Type":"ContainerDied","Data":"a8d807436980b40044176302736326635dd04bb27329ef6a25ffe8086e2f19c1"}
Sep 16 18:52:25.439451 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.439427    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv" event={"ID":"987af540-96fe-4ac8-844c-3aa916cf50dc","Type":"ContainerDied","Data":"55b6f5c510577284001cee42807a4dc5aef4bef4d19f97a5e43f288c55b4ae09"}
Sep 16 18:52:25.439573 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.439454    2554 scope.go:117] "RemoveContainer" containerID="a8d807436980b40044176302736326635dd04bb27329ef6a25ffe8086e2f19c1"
Sep 16 18:52:25.452974 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.452952    2554 scope.go:117] "RemoveContainer" containerID="31e5efa17837df162c29c0183b387ffd0b98e5a5f42f3aa340da6b7c748466b9"
Sep 16 18:52:25.461060 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.460953    2554 scope.go:117] "RemoveContainer" containerID="a8d807436980b40044176302736326635dd04bb27329ef6a25ffe8086e2f19c1"
Sep 16 18:52:25.461553 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:52:25.461528    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d807436980b40044176302736326635dd04bb27329ef6a25ffe8086e2f19c1\": container with ID starting with a8d807436980b40044176302736326635dd04bb27329ef6a25ffe8086e2f19c1 not found: ID does not exist" containerID="a8d807436980b40044176302736326635dd04bb27329ef6a25ffe8086e2f19c1"
Sep 16 18:52:25.461643 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.461558    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d807436980b40044176302736326635dd04bb27329ef6a25ffe8086e2f19c1"} err="failed to get container status \"a8d807436980b40044176302736326635dd04bb27329ef6a25ffe8086e2f19c1\": rpc error: code = NotFound desc = could not find container \"a8d807436980b40044176302736326635dd04bb27329ef6a25ffe8086e2f19c1\": container with ID starting with a8d807436980b40044176302736326635dd04bb27329ef6a25ffe8086e2f19c1 not found: ID does not exist"
Sep 16 18:52:25.461643 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.461575    2554 scope.go:117] "RemoveContainer" containerID="31e5efa17837df162c29c0183b387ffd0b98e5a5f42f3aa340da6b7c748466b9"
Sep 16 18:52:25.461852 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:52:25.461835    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e5efa17837df162c29c0183b387ffd0b98e5a5f42f3aa340da6b7c748466b9\": container with ID starting with 31e5efa17837df162c29c0183b387ffd0b98e5a5f42f3aa340da6b7c748466b9 not found: ID does not exist" containerID="31e5efa17837df162c29c0183b387ffd0b98e5a5f42f3aa340da6b7c748466b9"
Sep 16 18:52:25.461894 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.461857    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e5efa17837df162c29c0183b387ffd0b98e5a5f42f3aa340da6b7c748466b9"} err="failed to get container status \"31e5efa17837df162c29c0183b387ffd0b98e5a5f42f3aa340da6b7c748466b9\": rpc error: code = NotFound desc = could not find container \"31e5efa17837df162c29c0183b387ffd0b98e5a5f42f3aa340da6b7c748466b9\": container with ID starting with 31e5efa17837df162c29c0183b387ffd0b98e5a5f42f3aa340da6b7c748466b9 not found: ID does not exist"
Sep 16 18:52:25.463708 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.463687    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"]
Sep 16 18:52:25.467166 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.467145    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6817d3-openshift-default-7587cb98bd-5dpvv"]
Sep 16 18:52:25.504031 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.504005    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:25.504031 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.504029    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2c5r\" (UniqueName: \"kubernetes.io/projected/987af540-96fe-4ac8-844c-3aa916cf50dc-kube-api-access-k2c5r\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:25.504161 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.504039    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:25.504161 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.504048    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:25.504161 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.504056    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:25.504161 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.504063    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:25.504161 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:25.504070    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/987af540-96fe-4ac8-844c-3aa916cf50dc-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:26.884150 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:26.884116    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="987af540-96fe-4ac8-844c-3aa916cf50dc" path="/var/lib/kubelet/pods/987af540-96fe-4ac8-844c-3aa916cf50dc/volumes"
Sep 16 18:52:31.251672 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.251625    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"]
Sep 16 18:52:31.252126 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.251866    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9" podUID="c4213288-7ddc-40d7-bfa2-4b929bef4703" containerName="maas-api" containerID="cri-o://1722f5387a0429c94737c01f429f998e96defc8e6f784657b0d2c7947265ab83" gracePeriod=30
Sep 16 18:52:31.319067 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.319034    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"]
Sep 16 18:52:31.319279 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.319254    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4" podUID="6c73dd77-1850-4d0e-81df-0602437534fb" containerName="payload-processing" containerID="cri-o://523420269b2367080cee12efc1f6ef89283c5d36bd903aeba8b621015905528c" gracePeriod=30
Sep 16 18:52:31.340405 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.340372    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"]
Sep 16 18:52:31.340664 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.340618    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz" podUID="bef44bf0-e6e4-43af-801b-3ea0a13e061b" containerName="payload-pre-processing" containerID="cri-o://2ab8536ecdb1a64ab127d62d3c319d509bc5f5a03bac02cbf1793d61cd72835e" gracePeriod=30
Sep 16 18:52:31.472008 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.471976    2554 generic.go:356] "Generic (PLEG): container finished" podID="6c73dd77-1850-4d0e-81df-0602437534fb" containerID="523420269b2367080cee12efc1f6ef89283c5d36bd903aeba8b621015905528c" exitCode=0
Sep 16 18:52:31.472167 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.472085    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4" event={"ID":"6c73dd77-1850-4d0e-81df-0602437534fb","Type":"ContainerDied","Data":"523420269b2367080cee12efc1f6ef89283c5d36bd903aeba8b621015905528c"}
Sep 16 18:52:31.473975 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.473952    2554 generic.go:356] "Generic (PLEG): container finished" podID="bef44bf0-e6e4-43af-801b-3ea0a13e061b" containerID="2ab8536ecdb1a64ab127d62d3c319d509bc5f5a03bac02cbf1793d61cd72835e" exitCode=0
Sep 16 18:52:31.473975 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.474015    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz" event={"ID":"bef44bf0-e6e4-43af-801b-3ea0a13e061b","Type":"ContainerDied","Data":"2ab8536ecdb1a64ab127d62d3c319d509bc5f5a03bac02cbf1793d61cd72835e"}
Sep 16 18:52:31.476064 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.476022    2554 generic.go:356] "Generic (PLEG): container finished" podID="c4213288-7ddc-40d7-bfa2-4b929bef4703" containerID="1722f5387a0429c94737c01f429f998e96defc8e6f784657b0d2c7947265ab83" exitCode=0
Sep 16 18:52:31.476173 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.476107    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9" event={"ID":"c4213288-7ddc-40d7-bfa2-4b929bef4703","Type":"ContainerDied","Data":"1722f5387a0429c94737c01f429f998e96defc8e6f784657b0d2c7947265ab83"}
Sep 16 18:52:31.634264 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.634219    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"
Sep 16 18:52:31.637283 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.637262    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"
Sep 16 18:52:31.640704 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.640681    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:52:31.666684 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.666644    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bef44bf0-e6e4-43af-801b-3ea0a13e061b-plugins-config-volume\") pod \"bef44bf0-e6e4-43af-801b-3ea0a13e061b\" (UID: \"bef44bf0-e6e4-43af-801b-3ea0a13e061b\") "
Sep 16 18:52:31.666825 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.666705    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsp6h\" (UniqueName: \"kubernetes.io/projected/6c73dd77-1850-4d0e-81df-0602437534fb-kube-api-access-lsp6h\") pod \"6c73dd77-1850-4d0e-81df-0602437534fb\" (UID: \"6c73dd77-1850-4d0e-81df-0602437534fb\") "
Sep 16 18:52:31.666825 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.666749    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c73dd77-1850-4d0e-81df-0602437534fb-plugins-config-volume\") pod \"6c73dd77-1850-4d0e-81df-0602437534fb\" (UID: \"6c73dd77-1850-4d0e-81df-0602437534fb\") "
Sep 16 18:52:31.666825 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.666778    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4213288-7ddc-40d7-bfa2-4b929bef4703-metrics-certs\") pod \"c4213288-7ddc-40d7-bfa2-4b929bef4703\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") "
Sep 16 18:52:31.666825 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.666806    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4213288-7ddc-40d7-bfa2-4b929bef4703-maas-api-tls\") pod \"c4213288-7ddc-40d7-bfa2-4b929bef4703\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") "
Sep 16 18:52:31.667043 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.666836    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfl44\" (UniqueName: \"kubernetes.io/projected/bef44bf0-e6e4-43af-801b-3ea0a13e061b-kube-api-access-rfl44\") pod \"bef44bf0-e6e4-43af-801b-3ea0a13e061b\" (UID: \"bef44bf0-e6e4-43af-801b-3ea0a13e061b\") "
Sep 16 18:52:31.667043 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.666871    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4213288-7ddc-40d7-bfa2-4b929bef4703-openshift-service-ca\") pod \"c4213288-7ddc-40d7-bfa2-4b929bef4703\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") "
Sep 16 18:52:31.667043 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.666905    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqz87\" (UniqueName: \"kubernetes.io/projected/c4213288-7ddc-40d7-bfa2-4b929bef4703-kube-api-access-sqz87\") pod \"c4213288-7ddc-40d7-bfa2-4b929bef4703\" (UID: \"c4213288-7ddc-40d7-bfa2-4b929bef4703\") "
Sep 16 18:52:31.667142 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.667067    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef44bf0-e6e4-43af-801b-3ea0a13e061b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bef44bf0-e6e4-43af-801b-3ea0a13e061b" (UID: "bef44bf0-e6e4-43af-801b-3ea0a13e061b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:31.667203 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.667136    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c73dd77-1850-4d0e-81df-0602437534fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c73dd77-1850-4d0e-81df-0602437534fb" (UID: "6c73dd77-1850-4d0e-81df-0602437534fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:31.667314 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.667295    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bef44bf0-e6e4-43af-801b-3ea0a13e061b-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:31.667374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.667316    2554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c73dd77-1850-4d0e-81df-0602437534fb-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:31.667784 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.667750    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4213288-7ddc-40d7-bfa2-4b929bef4703-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c4213288-7ddc-40d7-bfa2-4b929bef4703" (UID: "c4213288-7ddc-40d7-bfa2-4b929bef4703"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:31.669461 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.669432    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4213288-7ddc-40d7-bfa2-4b929bef4703-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c4213288-7ddc-40d7-bfa2-4b929bef4703" (UID: "c4213288-7ddc-40d7-bfa2-4b929bef4703"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:52:31.669542 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.669436    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4213288-7ddc-40d7-bfa2-4b929bef4703-kube-api-access-sqz87" (OuterVolumeSpecName: "kube-api-access-sqz87") pod "c4213288-7ddc-40d7-bfa2-4b929bef4703" (UID: "c4213288-7ddc-40d7-bfa2-4b929bef4703"). InnerVolumeSpecName "kube-api-access-sqz87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:31.669761 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.669733    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef44bf0-e6e4-43af-801b-3ea0a13e061b-kube-api-access-rfl44" (OuterVolumeSpecName: "kube-api-access-rfl44") pod "bef44bf0-e6e4-43af-801b-3ea0a13e061b" (UID: "bef44bf0-e6e4-43af-801b-3ea0a13e061b"). InnerVolumeSpecName "kube-api-access-rfl44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:31.669907 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.669891    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4213288-7ddc-40d7-bfa2-4b929bef4703-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c4213288-7ddc-40d7-bfa2-4b929bef4703" (UID: "c4213288-7ddc-40d7-bfa2-4b929bef4703"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:52:31.669982 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.669958    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c73dd77-1850-4d0e-81df-0602437534fb-kube-api-access-lsp6h" (OuterVolumeSpecName: "kube-api-access-lsp6h") pod "6c73dd77-1850-4d0e-81df-0602437534fb" (UID: "6c73dd77-1850-4d0e-81df-0602437534fb"). InnerVolumeSpecName "kube-api-access-lsp6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:31.768764 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.768667    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsp6h\" (UniqueName: \"kubernetes.io/projected/6c73dd77-1850-4d0e-81df-0602437534fb-kube-api-access-lsp6h\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:31.768764 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.768700    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4213288-7ddc-40d7-bfa2-4b929bef4703-metrics-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:31.768764 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.768715    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4213288-7ddc-40d7-bfa2-4b929bef4703-maas-api-tls\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:31.768764 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.768727    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfl44\" (UniqueName: \"kubernetes.io/projected/bef44bf0-e6e4-43af-801b-3ea0a13e061b-kube-api-access-rfl44\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:31.768764 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.768738    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4213288-7ddc-40d7-bfa2-4b929bef4703-openshift-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:31.768764 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:31.768750    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqz87\" (UniqueName: \"kubernetes.io/projected/c4213288-7ddc-40d7-bfa2-4b929bef4703-kube-api-access-sqz87\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:32.482870 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.482838    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"
Sep 16 18:52:32.483334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.482873    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4" event={"ID":"6c73dd77-1850-4d0e-81df-0602437534fb","Type":"ContainerDied","Data":"83f378265590f72f153af774b2fa5a0cd953d081f8f4c709e40887e837bc0b01"}
Sep 16 18:52:32.483334 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.482934    2554 scope.go:117] "RemoveContainer" containerID="523420269b2367080cee12efc1f6ef89283c5d36bd903aeba8b621015905528c"
Sep 16 18:52:32.484295 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.484258    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz" event={"ID":"bef44bf0-e6e4-43af-801b-3ea0a13e061b","Type":"ContainerDied","Data":"4e4a18b312e8c0f588d15bf1ebee02c44c725bd53daec3278bbd04f049e70da6"}
Sep 16 18:52:32.484295 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.484280    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"
Sep 16 18:52:32.485773 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.485750    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"
Sep 16 18:52:32.485773 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.485762    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9" event={"ID":"c4213288-7ddc-40d7-bfa2-4b929bef4703","Type":"ContainerDied","Data":"7c72498088fff2f41035af42a81634d515f76a002f50b3e6cccca56d0f50dc98"}
Sep 16 18:52:32.493936 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.493723    2554 scope.go:117] "RemoveContainer" containerID="2ab8536ecdb1a64ab127d62d3c319d509bc5f5a03bac02cbf1793d61cd72835e"
Sep 16 18:52:32.508138 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.508115    2554 scope.go:117] "RemoveContainer" containerID="1722f5387a0429c94737c01f429f998e96defc8e6f784657b0d2c7947265ab83"
Sep 16 18:52:32.515342 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.515312    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"]
Sep 16 18:52:32.518764 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.518714    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c90bd2-686c5d567f-c4zb4"]
Sep 16 18:52:32.530484 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.530460    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"]
Sep 16 18:52:32.536175 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.536152    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c90bd2-85c55f84b6-6mvlz"]
Sep 16 18:52:32.567638 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.567607    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"]
Sep 16 18:52:32.573854 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.573832    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c90bd2-56fbbd5c55-6vbd9"]
Sep 16 18:52:32.886467 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.886381    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c73dd77-1850-4d0e-81df-0602437534fb" path="/var/lib/kubelet/pods/6c73dd77-1850-4d0e-81df-0602437534fb/volumes"
Sep 16 18:52:32.886868 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.886847    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bef44bf0-e6e4-43af-801b-3ea0a13e061b" path="/var/lib/kubelet/pods/bef44bf0-e6e4-43af-801b-3ea0a13e061b/volumes"
Sep 16 18:52:32.887295 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:32.887277    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4213288-7ddc-40d7-bfa2-4b929bef4703" path="/var/lib/kubelet/pods/c4213288-7ddc-40d7-bfa2-4b929bef4703/volumes"
Sep 16 18:52:34.611356 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:34.611318    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:52:49.247226 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.247183    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"]
Sep 16 18:52:49.247863 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.247478    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg" podUID="24d554af-5c77-4adc-8ef9-f49e6b184ac2" containerName="maas-api" containerID="cri-o://caaba526d877c25213213541a28a60cb7f0a95de0c0901f4600d70ccf8d1954e" gracePeriod=30
Sep 16 18:52:49.491401 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.491375    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:52:49.576844 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.576755    2554 generic.go:356] "Generic (PLEG): container finished" podID="24d554af-5c77-4adc-8ef9-f49e6b184ac2" containerID="caaba526d877c25213213541a28a60cb7f0a95de0c0901f4600d70ccf8d1954e" exitCode=0
Sep 16 18:52:49.576844 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.576797    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"
Sep 16 18:52:49.576844 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.576835    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg" event={"ID":"24d554af-5c77-4adc-8ef9-f49e6b184ac2","Type":"ContainerDied","Data":"caaba526d877c25213213541a28a60cb7f0a95de0c0901f4600d70ccf8d1954e"}
Sep 16 18:52:49.577066 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.576886    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg" event={"ID":"24d554af-5c77-4adc-8ef9-f49e6b184ac2","Type":"ContainerDied","Data":"244c9b40a22cabef55becfbfa7934355b63867d45b526a85c5b93e6427625597"}
Sep 16 18:52:49.577066 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.576908    2554 scope.go:117] "RemoveContainer" containerID="caaba526d877c25213213541a28a60cb7f0a95de0c0901f4600d70ccf8d1954e"
Sep 16 18:52:49.585722 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.585703    2554 scope.go:117] "RemoveContainer" containerID="caaba526d877c25213213541a28a60cb7f0a95de0c0901f4600d70ccf8d1954e"
Sep 16 18:52:49.585963 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:52:49.585946    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caaba526d877c25213213541a28a60cb7f0a95de0c0901f4600d70ccf8d1954e\": container with ID starting with caaba526d877c25213213541a28a60cb7f0a95de0c0901f4600d70ccf8d1954e not found: ID does not exist" containerID="caaba526d877c25213213541a28a60cb7f0a95de0c0901f4600d70ccf8d1954e"
Sep 16 18:52:49.586004 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.585970    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caaba526d877c25213213541a28a60cb7f0a95de0c0901f4600d70ccf8d1954e"} err="failed to get container status \"caaba526d877c25213213541a28a60cb7f0a95de0c0901f4600d70ccf8d1954e\": rpc error: code = NotFound desc = could not find container \"caaba526d877c25213213541a28a60cb7f0a95de0c0901f4600d70ccf8d1954e\": container with ID starting with caaba526d877c25213213541a28a60cb7f0a95de0c0901f4600d70ccf8d1954e not found: ID does not exist"
Sep 16 18:52:49.632036 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.632006    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2c5j\" (UniqueName: \"kubernetes.io/projected/24d554af-5c77-4adc-8ef9-f49e6b184ac2-kube-api-access-h2c5j\") pod \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") "
Sep 16 18:52:49.632213 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.632074    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24d554af-5c77-4adc-8ef9-f49e6b184ac2-maas-api-tls\") pod \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") "
Sep 16 18:52:49.632213 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.632116    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24d554af-5c77-4adc-8ef9-f49e6b184ac2-metrics-certs\") pod \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") "
Sep 16 18:52:49.632213 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.632134    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d554af-5c77-4adc-8ef9-f49e6b184ac2-openshift-service-ca\") pod \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\" (UID: \"24d554af-5c77-4adc-8ef9-f49e6b184ac2\") "
Sep 16 18:52:49.632533 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.632500    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d554af-5c77-4adc-8ef9-f49e6b184ac2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24d554af-5c77-4adc-8ef9-f49e6b184ac2" (UID: "24d554af-5c77-4adc-8ef9-f49e6b184ac2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:49.634196 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.634170    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d554af-5c77-4adc-8ef9-f49e6b184ac2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "24d554af-5c77-4adc-8ef9-f49e6b184ac2" (UID: "24d554af-5c77-4adc-8ef9-f49e6b184ac2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:52:49.634306 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.634199    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d554af-5c77-4adc-8ef9-f49e6b184ac2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24d554af-5c77-4adc-8ef9-f49e6b184ac2" (UID: "24d554af-5c77-4adc-8ef9-f49e6b184ac2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:52:49.634306 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.634199    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d554af-5c77-4adc-8ef9-f49e6b184ac2-kube-api-access-h2c5j" (OuterVolumeSpecName: "kube-api-access-h2c5j") pod "24d554af-5c77-4adc-8ef9-f49e6b184ac2" (UID: "24d554af-5c77-4adc-8ef9-f49e6b184ac2"). InnerVolumeSpecName "kube-api-access-h2c5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:49.733096 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.733046    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24d554af-5c77-4adc-8ef9-f49e6b184ac2-maas-api-tls\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:49.733096 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.733090    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24d554af-5c77-4adc-8ef9-f49e6b184ac2-metrics-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:49.733096 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.733099    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d554af-5c77-4adc-8ef9-f49e6b184ac2-openshift-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:49.733096 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.733109    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2c5j\" (UniqueName: \"kubernetes.io/projected/24d554af-5c77-4adc-8ef9-f49e6b184ac2-kube-api-access-h2c5j\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:52:49.898755 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.898715    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"]
Sep 16 18:52:49.902722 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:49.902696    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-972df0-6c6c7d8c87-dzsbg"]
Sep 16 18:52:50.884505 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:50.884472    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24d554af-5c77-4adc-8ef9-f49e6b184ac2" path="/var/lib/kubelet/pods/24d554af-5c77-4adc-8ef9-f49e6b184ac2/volumes"
Sep 16 18:52:52.500869 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:52.500827    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:52:56.989872 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.989836    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x"]
Sep 16 18:52:56.990225 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990190    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987af540-96fe-4ac8-844c-3aa916cf50dc" containerName="istio-proxy"
Sep 16 18:52:56.990225 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990201    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="987af540-96fe-4ac8-844c-3aa916cf50dc" containerName="istio-proxy"
Sep 16 18:52:56.990225 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990218    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55270654-9067-49d3-8b75-f249da99d76b" containerName="payload-processing"
Sep 16 18:52:56.990225 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990225    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="55270654-9067-49d3-8b75-f249da99d76b" containerName="payload-processing"
Sep 16 18:52:56.990384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990247    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="703190ae-ec9d-44f3-893c-d67518deed1c" containerName="payload-pre-processing"
Sep 16 18:52:56.990384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990254    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="703190ae-ec9d-44f3-893c-d67518deed1c" containerName="payload-pre-processing"
Sep 16 18:52:56.990384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990262    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4213288-7ddc-40d7-bfa2-4b929bef4703" containerName="maas-api"
Sep 16 18:52:56.990384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990267    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4213288-7ddc-40d7-bfa2-4b929bef4703" containerName="maas-api"
Sep 16 18:52:56.990384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990275    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24d554af-5c77-4adc-8ef9-f49e6b184ac2" containerName="maas-api"
Sep 16 18:52:56.990384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990279    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="24d554af-5c77-4adc-8ef9-f49e6b184ac2" containerName="maas-api"
Sep 16 18:52:56.990384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990287    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2170aee3-5439-489c-a94b-c5b46663666c" containerName="maas-api"
Sep 16 18:52:56.990384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990292    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2170aee3-5439-489c-a94b-c5b46663666c" containerName="maas-api"
Sep 16 18:52:56.990384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990298    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c73dd77-1850-4d0e-81df-0602437534fb" containerName="payload-processing"
Sep 16 18:52:56.990384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990303    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c73dd77-1850-4d0e-81df-0602437534fb" containerName="payload-processing"
Sep 16 18:52:56.990384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990318    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef44bf0-e6e4-43af-801b-3ea0a13e061b" containerName="payload-pre-processing"
Sep 16 18:52:56.990384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990323    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef44bf0-e6e4-43af-801b-3ea0a13e061b" containerName="payload-pre-processing"
Sep 16 18:52:56.990384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990375    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bef44bf0-e6e4-43af-801b-3ea0a13e061b" containerName="payload-pre-processing"
Sep 16 18:52:56.990384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990381    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c73dd77-1850-4d0e-81df-0602437534fb" containerName="payload-processing"
Sep 16 18:52:56.990384 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990388    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="987af540-96fe-4ac8-844c-3aa916cf50dc" containerName="istio-proxy"
Sep 16 18:52:56.990810 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990394    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="24d554af-5c77-4adc-8ef9-f49e6b184ac2" containerName="maas-api"
Sep 16 18:52:56.990810 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990400    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4213288-7ddc-40d7-bfa2-4b929bef4703" containerName="maas-api"
Sep 16 18:52:56.990810 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990406    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="703190ae-ec9d-44f3-893c-d67518deed1c" containerName="payload-pre-processing"
Sep 16 18:52:56.990810 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990414    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2170aee3-5439-489c-a94b-c5b46663666c" containerName="maas-api"
Sep 16 18:52:56.990810 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.990419    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="55270654-9067-49d3-8b75-f249da99d76b" containerName="payload-processing"
Sep 16 18:52:56.993472 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.993455    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x"
Sep 16 18:52:56.995807 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:56.995789    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lfw6p\""
Sep 16 18:52:57.002081 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:57.002060    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x"]
Sep 16 18:52:57.098600 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:57.098556    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e3c484-b51d-402e-97d2-fd60e9f7b6ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x\" (UID: \"71e3c484-b51d-402e-97d2-fd60e9f7b6ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x"
Sep 16 18:52:57.199697 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:57.199641    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e3c484-b51d-402e-97d2-fd60e9f7b6ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x\" (UID: \"71e3c484-b51d-402e-97d2-fd60e9f7b6ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x"
Sep 16 18:52:57.200310 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:57.200290    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e3c484-b51d-402e-97d2-fd60e9f7b6ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x\" (UID: \"71e3c484-b51d-402e-97d2-fd60e9f7b6ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x"
Sep 16 18:52:57.304331 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:57.304223    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x"
Sep 16 18:52:57.575631 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:57.575604    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x"]
Sep 16 18:52:57.578306 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:52:57.578277    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e3c484_b51d_402e_97d2_fd60e9f7b6ec.slice/crio-0e201332def76e1bc78980f1605d580567c83e0379fb93833f63646f51e706a2 WatchSource:0}: Error finding container 0e201332def76e1bc78980f1605d580567c83e0379fb93833f63646f51e706a2: Status 404 returned error can't find the container with id 0e201332def76e1bc78980f1605d580567c83e0379fb93833f63646f51e706a2
Sep 16 18:52:57.616826 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:57.616791    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x" event={"ID":"71e3c484-b51d-402e-97d2-fd60e9f7b6ec","Type":"ContainerStarted","Data":"0e201332def76e1bc78980f1605d580567c83e0379fb93833f63646f51e706a2"}
Sep 16 18:52:58.623734 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:58.623636    2554 generic.go:356] "Generic (PLEG): container finished" podID="71e3c484-b51d-402e-97d2-fd60e9f7b6ec" containerID="bb5453f21d78ee8dcdeb3e94f88942aa0b8bfb8102284ad14a7a56bc042473b6" exitCode=0
Sep 16 18:52:58.623734 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:58.623718    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x" event={"ID":"71e3c484-b51d-402e-97d2-fd60e9f7b6ec","Type":"ContainerDied","Data":"bb5453f21d78ee8dcdeb3e94f88942aa0b8bfb8102284ad14a7a56bc042473b6"}
Sep 16 18:52:59.761931 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:59.761908    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x"
Sep 16 18:52:59.924483 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:59.924372    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e3c484-b51d-402e-97d2-fd60e9f7b6ec-maas-api-service-ca\") pod \"71e3c484-b51d-402e-97d2-fd60e9f7b6ec\" (UID: \"71e3c484-b51d-402e-97d2-fd60e9f7b6ec\") "
Sep 16 18:52:59.924832 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:52:59.924806    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e3c484-b51d-402e-97d2-fd60e9f7b6ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "71e3c484-b51d-402e-97d2-fd60e9f7b6ec" (UID: "71e3c484-b51d-402e-97d2-fd60e9f7b6ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:53:00.025424 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:53:00.025394    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e3c484-b51d-402e-97d2-fd60e9f7b6ec-maas-api-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:53:00.636185 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:53:00.636157    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x"
Sep 16 18:53:00.636383 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:53:00.636160    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x" event={"ID":"71e3c484-b51d-402e-97d2-fd60e9f7b6ec","Type":"ContainerDied","Data":"0e201332def76e1bc78980f1605d580567c83e0379fb93833f63646f51e706a2"}
Sep 16 18:53:00.636383 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:53:00.636302    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e201332def76e1bc78980f1605d580567c83e0379fb93833f63646f51e706a2"
Sep 16 18:53:07.533483 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:53:07.533450    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d3e8be-29826405-lm4nq"]
Sep 16 18:53:07.535373 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:53:07.535350    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d3e8be-29826405-lm4nq"]
Sep 16 18:53:07.724590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:53:07.724549    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x"]
Sep 16 18:53:07.727071 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:53:07.727045    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d3e8be-9g84x"]
Sep 16 18:53:08.883576 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:53:08.883545    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="491a823f-3e5c-480a-98e0-6c40fd6cd562" path="/var/lib/kubelet/pods/491a823f-3e5c-480a-98e0-6c40fd6cd562/volumes"
Sep 16 18:53:08.883942 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:53:08.883846    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71e3c484-b51d-402e-97d2-fd60e9f7b6ec" path="/var/lib/kubelet/pods/71e3c484-b51d-402e-97d2-fd60e9f7b6ec/volumes"
Sep 16 18:53:10.114044 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:53:10.114003    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:53:17.233378 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:53:17.233344    2554 scope.go:117] "RemoveContainer" containerID="713c04e6cf30ca64b675e8866bf6631be37c47c1085150f9eedfa32dd50980eb"
Sep 16 18:53:17.241537 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:53:17.241508    2554 scope.go:117] "RemoveContainer" containerID="fca3c4c41c5832921ceecabe326a80a85a78dc4586c38d04cb643618f61b5c76"
Sep 16 18:53:20.301928 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:53:20.301892    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:54:41.109554 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:54:41.109512    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:55:05.105684 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:05.105651    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:55:12.749213 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:12.749124    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:55:22.105117 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:22.105081    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:55:29.308189 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:29.308155    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:55:41.014517 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:41.014479    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:55:46.699834 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:46.699793    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:55:48.521374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:48.521334    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"]
Sep 16 18:55:48.521852 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:48.521593    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln" podUID="f5313a4b-6151-4a66-9688-b4c6b29f3a95" containerName="manager" containerID="cri-o://93f36ec90b5acfb315db60d93b76ec1e101b6e9ae964f407a45984cb5e3e75df" gracePeriod=10
Sep 16 18:55:50.479864 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:50.479835    2554 generic.go:356] "Generic (PLEG): container finished" podID="f5313a4b-6151-4a66-9688-b4c6b29f3a95" containerID="93f36ec90b5acfb315db60d93b76ec1e101b6e9ae964f407a45984cb5e3e75df" exitCode=0
Sep 16 18:55:50.480271 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:50.479904    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln" event={"ID":"f5313a4b-6151-4a66-9688-b4c6b29f3a95","Type":"ContainerDied","Data":"93f36ec90b5acfb315db60d93b76ec1e101b6e9ae964f407a45984cb5e3e75df"}
Sep 16 18:55:50.480271 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:50.479963    2554 scope.go:117] "RemoveContainer" containerID="18ac9806dc04f192cf34ccfeced8bae480a03a74837d54c22d2f4249ac0aeb1b"
Sep 16 18:55:50.772181 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:50.772154    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"
Sep 16 18:55:50.857898 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:50.857862    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b82qr\" (UniqueName: \"kubernetes.io/projected/f5313a4b-6151-4a66-9688-b4c6b29f3a95-kube-api-access-b82qr\") pod \"f5313a4b-6151-4a66-9688-b4c6b29f3a95\" (UID: \"f5313a4b-6151-4a66-9688-b4c6b29f3a95\") "
Sep 16 18:55:50.858090 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:50.857990    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5313a4b-6151-4a66-9688-b4c6b29f3a95-extensions-socket-volume\") pod \"f5313a4b-6151-4a66-9688-b4c6b29f3a95\" (UID: \"f5313a4b-6151-4a66-9688-b4c6b29f3a95\") "
Sep 16 18:55:50.858351 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:50.858329    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5313a4b-6151-4a66-9688-b4c6b29f3a95-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f5313a4b-6151-4a66-9688-b4c6b29f3a95" (UID: "f5313a4b-6151-4a66-9688-b4c6b29f3a95"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:50.859899 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:50.859879    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5313a4b-6151-4a66-9688-b4c6b29f3a95-kube-api-access-b82qr" (OuterVolumeSpecName: "kube-api-access-b82qr") pod "f5313a4b-6151-4a66-9688-b4c6b29f3a95" (UID: "f5313a4b-6151-4a66-9688-b4c6b29f3a95"). InnerVolumeSpecName "kube-api-access-b82qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:50.959602 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:50.959562    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b82qr\" (UniqueName: \"kubernetes.io/projected/f5313a4b-6151-4a66-9688-b4c6b29f3a95-kube-api-access-b82qr\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:55:50.959602 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:50.959593    2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5313a4b-6151-4a66-9688-b4c6b29f3a95-extensions-socket-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:55:51.485934 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:51.485900    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"
Sep 16 18:55:51.486452 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:51.485937    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln" event={"ID":"f5313a4b-6151-4a66-9688-b4c6b29f3a95","Type":"ContainerDied","Data":"66c9ef183e286738b5949ab4e8c84ec375e328186352ca2ca6ec3d8f5e6ea895"}
Sep 16 18:55:51.486452 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:51.485993    2554 scope.go:117] "RemoveContainer" containerID="93f36ec90b5acfb315db60d93b76ec1e101b6e9ae964f407a45984cb5e3e75df"
Sep 16 18:55:51.510822 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:51.510757    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"]
Sep 16 18:55:51.512689 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:51.512667    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-pqhln"]
Sep 16 18:55:52.883790 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:55:52.883755    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5313a4b-6151-4a66-9688-b4c6b29f3a95" path="/var/lib/kubelet/pods/f5313a4b-6151-4a66-9688-b4c6b29f3a95/volumes"
Sep 16 18:56:16.990213 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:16.990186    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:56:17.001684 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:17.001658    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:56:49.339082 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.339044    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks8vt"]
Sep 16 18:56:49.339572 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.339425    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5313a4b-6151-4a66-9688-b4c6b29f3a95" containerName="manager"
Sep 16 18:56:49.339572 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.339437    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5313a4b-6151-4a66-9688-b4c6b29f3a95" containerName="manager"
Sep 16 18:56:49.339572 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.339451    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e3c484-b51d-402e-97d2-fd60e9f7b6ec" containerName="revoke-keys"
Sep 16 18:56:49.339572 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.339458    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e3c484-b51d-402e-97d2-fd60e9f7b6ec" containerName="revoke-keys"
Sep 16 18:56:49.339572 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.339471    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987af540-96fe-4ac8-844c-3aa916cf50dc" containerName="istio-proxy"
Sep 16 18:56:49.339572 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.339477    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="987af540-96fe-4ac8-844c-3aa916cf50dc" containerName="istio-proxy"
Sep 16 18:56:49.339572 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.339528    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="987af540-96fe-4ac8-844c-3aa916cf50dc" containerName="istio-proxy"
Sep 16 18:56:49.339572 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.339536    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5313a4b-6151-4a66-9688-b4c6b29f3a95" containerName="manager"
Sep 16 18:56:49.339572 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.339543    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e3c484-b51d-402e-97d2-fd60e9f7b6ec" containerName="revoke-keys"
Sep 16 18:56:49.339572 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.339551    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5313a4b-6151-4a66-9688-b4c6b29f3a95" containerName="manager"
Sep 16 18:56:49.342483 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.342461    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks8vt"
Sep 16 18:56:49.354476 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.354459    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6bv26\""
Sep 16 18:56:49.385047 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.385017    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks8vt"]
Sep 16 18:56:49.462456 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.462423    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn9wr\" (UniqueName: \"kubernetes.io/projected/ce11585d-55c4-476c-8e3d-aad3e662ed4c-kube-api-access-pn9wr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ks8vt\" (UID: \"ce11585d-55c4-476c-8e3d-aad3e662ed4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks8vt"
Sep 16 18:56:49.462616 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.462474    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce11585d-55c4-476c-8e3d-aad3e662ed4c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ks8vt\" (UID: \"ce11585d-55c4-476c-8e3d-aad3e662ed4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks8vt"
Sep 16 18:56:49.563319 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.563282    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn9wr\" (UniqueName: \"kubernetes.io/projected/ce11585d-55c4-476c-8e3d-aad3e662ed4c-kube-api-access-pn9wr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ks8vt\" (UID: \"ce11585d-55c4-476c-8e3d-aad3e662ed4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks8vt"
Sep 16 18:56:49.563486 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.563336    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce11585d-55c4-476c-8e3d-aad3e662ed4c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ks8vt\" (UID: \"ce11585d-55c4-476c-8e3d-aad3e662ed4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks8vt"
Sep 16 18:56:49.563656 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.563641    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce11585d-55c4-476c-8e3d-aad3e662ed4c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ks8vt\" (UID: \"ce11585d-55c4-476c-8e3d-aad3e662ed4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks8vt"
Sep 16 18:56:49.573634 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.573615    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn9wr\" (UniqueName: \"kubernetes.io/projected/ce11585d-55c4-476c-8e3d-aad3e662ed4c-kube-api-access-pn9wr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ks8vt\" (UID: \"ce11585d-55c4-476c-8e3d-aad3e662ed4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks8vt"
Sep 16 18:56:49.651966 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.651873    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks8vt"
Sep 16 18:56:49.934951 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:49.934871    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks8vt"]
Sep 16 18:56:49.938650 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:56:49.938616    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce11585d_55c4_476c_8e3d_aad3e662ed4c.slice/crio-aa22d52bd9ae9b5c0b3fa0842b6549ec0b11acdf7edd4d6a8bacb2a8497df590 WatchSource:0}: Error finding container aa22d52bd9ae9b5c0b3fa0842b6549ec0b11acdf7edd4d6a8bacb2a8497df590: Status 404 returned error can't find the container with id aa22d52bd9ae9b5c0b3fa0842b6549ec0b11acdf7edd4d6a8bacb2a8497df590
Sep 16 18:56:50.772453 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:50.772410    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks8vt" event={"ID":"ce11585d-55c4-476c-8e3d-aad3e662ed4c","Type":"ContainerStarted","Data":"0bc87f9f6dc5c3e9f7362974edfc9a8dd2f0450a220a918e639dec7ab104c4b1"}
Sep 16 18:56:50.772824 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:50.772476    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks8vt" event={"ID":"ce11585d-55c4-476c-8e3d-aad3e662ed4c","Type":"ContainerStarted","Data":"aa22d52bd9ae9b5c0b3fa0842b6549ec0b11acdf7edd4d6a8bacb2a8497df590"}
Sep 16 18:56:50.772824 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:50.772563    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks8vt"
Sep 16 18:56:50.798969 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:56:50.798903    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks8vt" podStartSLOduration=1.798885436 podStartE2EDuration="1.798885436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:50.796823475 +0000 UTC m=+2434.355991307" watchObservedRunningTime="2026-09-16 18:56:50.798885436 +0000 UTC m=+2434.358053270"
Sep 16 18:57:01.780221 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:01.780186    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks8vt"
Sep 16 18:57:17.315303 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:17.315270    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:57:23.243353 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:23.243318    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:57:37.114776 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.114737    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"]
Sep 16 18:57:37.115363 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.115343    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5313a4b-6151-4a66-9688-b4c6b29f3a95" containerName="manager"
Sep 16 18:57:37.115467 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.115366    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5313a4b-6151-4a66-9688-b4c6b29f3a95" containerName="manager"
Sep 16 18:57:37.118505 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.118481    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.122785 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.122756    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-624f44dc-kserve-self-signed-certs\""
Sep 16 18:57:37.132524 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.132497    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"]
Sep 16 18:57:37.194827 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.194792    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-tls-certs\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.194827 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.194825    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-model-cache\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.195030 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.194874    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-tmp-dir\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.195030 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.194919    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfdvf\" (UniqueName: \"kubernetes.io/projected/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-kube-api-access-lfdvf\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.195030 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.194962    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-home\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.195030 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.194976    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-dshm\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.296181 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.296141    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-tls-certs\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.296181 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.296180    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-model-cache\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.296499 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.296250    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-tmp-dir\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.296499 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.296277    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfdvf\" (UniqueName: \"kubernetes.io/projected/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-kube-api-access-lfdvf\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.296499 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.296326    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-home\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.296499 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.296366    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-dshm\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.296742 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.296714    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-model-cache\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.296797 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.296745    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-tmp-dir\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.296797 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.296769    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-home\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.298601 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.298582    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-dshm\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.298883 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.298863    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-tls-certs\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.305738 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.305712    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfdvf\" (UniqueName: \"kubernetes.io/projected/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-kube-api-access-lfdvf\") pod \"e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") " pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.429288 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.429176    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:37.709580 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:37.709496    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"]
Sep 16 18:57:37.715198 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:57:37.715168    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb21530c8_bf5e_4b10_8889_06b7c6b2ac7e.slice/crio-60a705588890f054432319d494b2ebbc69e7a72db9a2305cb14a5ef414122fb8 WatchSource:0}: Error finding container 60a705588890f054432319d494b2ebbc69e7a72db9a2305cb14a5ef414122fb8: Status 404 returned error can't find the container with id 60a705588890f054432319d494b2ebbc69e7a72db9a2305cb14a5ef414122fb8
Sep 16 18:57:38.007516 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:38.007467    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x" event={"ID":"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e","Type":"ContainerStarted","Data":"6b33094a081dc8e7584cbd2f5a5ffb74be081da5bdf84cead541245bd814fd8c"}
Sep 16 18:57:38.007701 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:38.007525    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x" event={"ID":"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e","Type":"ContainerStarted","Data":"60a705588890f054432319d494b2ebbc69e7a72db9a2305cb14a5ef414122fb8"}
Sep 16 18:57:38.033336 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:38.033286    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x" podStartSLOduration=1.033269294 podStartE2EDuration="1.033269294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:38.032261628 +0000 UTC m=+2481.591429462" watchObservedRunningTime="2026-09-16 18:57:38.033269294 +0000 UTC m=+2481.592437127"
Sep 16 18:57:39.710104 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:39.710066    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:57:47.429552 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:47.429514    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:47.429999 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:47.429567    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:47.441619 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:47.441592    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:48.071739 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:48.071712    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:57:54.119623 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:57:54.119586    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:58:03.922303 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:03.922270    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"]
Sep 16 18:58:07.125056 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:07.125009    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:58:14.530133 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:14.530047    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:58:18.940418 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:18.940349    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x" podUID="b21530c8-bf5e-4b10-8889-06b7c6b2ac7e" containerName="main" containerID="cri-o://6b33094a081dc8e7584cbd2f5a5ffb74be081da5bdf84cead541245bd814fd8c" gracePeriod=45
Sep 16 18:58:19.189640 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.189616    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:58:19.217874 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.217808    2554 generic.go:356] "Generic (PLEG): container finished" podID="b21530c8-bf5e-4b10-8889-06b7c6b2ac7e" containerID="6b33094a081dc8e7584cbd2f5a5ffb74be081da5bdf84cead541245bd814fd8c" exitCode=0
Sep 16 18:58:19.217874 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.217849    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"
Sep 16 18:58:19.217874 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.217855    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x" event={"ID":"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e","Type":"ContainerDied","Data":"6b33094a081dc8e7584cbd2f5a5ffb74be081da5bdf84cead541245bd814fd8c"}
Sep 16 18:58:19.218111 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.217892    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x" event={"ID":"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e","Type":"ContainerDied","Data":"60a705588890f054432319d494b2ebbc69e7a72db9a2305cb14a5ef414122fb8"}
Sep 16 18:58:19.218111 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.217915    2554 scope.go:117] "RemoveContainer" containerID="6b33094a081dc8e7584cbd2f5a5ffb74be081da5bdf84cead541245bd814fd8c"
Sep 16 18:58:19.227904 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.227877    2554 scope.go:117] "RemoveContainer" containerID="6b33094a081dc8e7584cbd2f5a5ffb74be081da5bdf84cead541245bd814fd8c"
Sep 16 18:58:19.228250 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:58:19.228204    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b33094a081dc8e7584cbd2f5a5ffb74be081da5bdf84cead541245bd814fd8c\": container with ID starting with 6b33094a081dc8e7584cbd2f5a5ffb74be081da5bdf84cead541245bd814fd8c not found: ID does not exist" containerID="6b33094a081dc8e7584cbd2f5a5ffb74be081da5bdf84cead541245bd814fd8c"
Sep 16 18:58:19.228327 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.228298    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b33094a081dc8e7584cbd2f5a5ffb74be081da5bdf84cead541245bd814fd8c"} err="failed to get container status \"6b33094a081dc8e7584cbd2f5a5ffb74be081da5bdf84cead541245bd814fd8c\": rpc error: code = NotFound desc = could not find container \"6b33094a081dc8e7584cbd2f5a5ffb74be081da5bdf84cead541245bd814fd8c\": container with ID starting with 6b33094a081dc8e7584cbd2f5a5ffb74be081da5bdf84cead541245bd814fd8c not found: ID does not exist"
Sep 16 18:58:19.245884 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.245861    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-tmp-dir\") pod \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") "
Sep 16 18:58:19.246018 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.245907    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-home\") pod \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") "
Sep 16 18:58:19.246018 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.245943    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-model-cache\") pod \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") "
Sep 16 18:58:19.246018 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.245986    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-dshm\") pod \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") "
Sep 16 18:58:19.246182 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.246063    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfdvf\" (UniqueName: \"kubernetes.io/projected/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-kube-api-access-lfdvf\") pod \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") "
Sep 16 18:58:19.246182 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.246107    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-tls-certs\") pod \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\" (UID: \"b21530c8-bf5e-4b10-8889-06b7c6b2ac7e\") "
Sep 16 18:58:19.246182 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.246135    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b21530c8-bf5e-4b10-8889-06b7c6b2ac7e" (UID: "b21530c8-bf5e-4b10-8889-06b7c6b2ac7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:19.246354 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.246213    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-home" (OuterVolumeSpecName: "home") pod "b21530c8-bf5e-4b10-8889-06b7c6b2ac7e" (UID: "b21530c8-bf5e-4b10-8889-06b7c6b2ac7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:19.246354 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.246226    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b21530c8-bf5e-4b10-8889-06b7c6b2ac7e" (UID: "b21530c8-bf5e-4b10-8889-06b7c6b2ac7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:19.246458 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.246419    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-tmp-dir\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:58:19.246458 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.246438    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-home\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:58:19.246458 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.246450    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-model-cache\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:58:19.248284 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.248259    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b21530c8-bf5e-4b10-8889-06b7c6b2ac7e" (UID: "b21530c8-bf5e-4b10-8889-06b7c6b2ac7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:19.248479 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.248459    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-dshm" (OuterVolumeSpecName: "dshm") pod "b21530c8-bf5e-4b10-8889-06b7c6b2ac7e" (UID: "b21530c8-bf5e-4b10-8889-06b7c6b2ac7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:19.248544 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.248497    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-kube-api-access-lfdvf" (OuterVolumeSpecName: "kube-api-access-lfdvf") pod "b21530c8-bf5e-4b10-8889-06b7c6b2ac7e" (UID: "b21530c8-bf5e-4b10-8889-06b7c6b2ac7e"). InnerVolumeSpecName "kube-api-access-lfdvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:19.347172 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.347136    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-dshm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:58:19.347172 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.347174    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfdvf\" (UniqueName: \"kubernetes.io/projected/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-kube-api-access-lfdvf\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:58:19.347394 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.347189    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e-tls-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:58:19.540604 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.540571    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"]
Sep 16 18:58:19.545461 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:19.545436    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-624f44dc-kserve-7774cbc9-wkw2x"]
Sep 16 18:58:20.883974 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:20.883934    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b21530c8-bf5e-4b10-8889-06b7c6b2ac7e" path="/var/lib/kubelet/pods/b21530c8-bf5e-4b10-8889-06b7c6b2ac7e/volumes"
Sep 16 18:58:22.216833 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.216795    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:58:22.309417 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.309367    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"]
Sep 16 18:58:22.309756 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.309745    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b21530c8-bf5e-4b10-8889-06b7c6b2ac7e" containerName="main"
Sep 16 18:58:22.309799 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.309758    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b21530c8-bf5e-4b10-8889-06b7c6b2ac7e" containerName="main"
Sep 16 18:58:22.309830 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.309813    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b21530c8-bf5e-4b10-8889-06b7c6b2ac7e" containerName="main"
Sep 16 18:58:22.314576 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.314556    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:22.317911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.317890    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 18:58:22.317911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.317904    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7l879\""
Sep 16 18:58:22.318062 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.317923    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 18:58:22.326895 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.326869    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"]
Sep 16 18:58:22.373799 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.373764    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-maas-api-tls\") pod \"maas-api-b66c78b8-8nzvh\" (UID: \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\") " pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:22.373939 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.373823    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-metrics-certs\") pod \"maas-api-b66c78b8-8nzvh\" (UID: \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\") " pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:22.373939 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.373881    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfchg\" (UniqueName: \"kubernetes.io/projected/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-kube-api-access-bfchg\") pod \"maas-api-b66c78b8-8nzvh\" (UID: \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\") " pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:22.373939 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.373899    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-openshift-service-ca\") pod \"maas-api-b66c78b8-8nzvh\" (UID: \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\") " pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:22.475131 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.475040    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-maas-api-tls\") pod \"maas-api-b66c78b8-8nzvh\" (UID: \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\") " pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:22.475131 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.475101    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-metrics-certs\") pod \"maas-api-b66c78b8-8nzvh\" (UID: \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\") " pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:22.475131 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.475131    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfchg\" (UniqueName: \"kubernetes.io/projected/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-kube-api-access-bfchg\") pod \"maas-api-b66c78b8-8nzvh\" (UID: \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\") " pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:22.475476 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.475147    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-openshift-service-ca\") pod \"maas-api-b66c78b8-8nzvh\" (UID: \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\") " pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:22.475867 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.475844    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-openshift-service-ca\") pod \"maas-api-b66c78b8-8nzvh\" (UID: \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\") " pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:22.477746 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.477723    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-maas-api-tls\") pod \"maas-api-b66c78b8-8nzvh\" (UID: \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\") " pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:22.477835 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.477723    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-metrics-certs\") pod \"maas-api-b66c78b8-8nzvh\" (UID: \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\") " pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:22.486539 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.486518    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfchg\" (UniqueName: \"kubernetes.io/projected/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-kube-api-access-bfchg\") pod \"maas-api-b66c78b8-8nzvh\" (UID: \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\") " pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:22.624870 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.624832    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:22.899979 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.899945    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"]
Sep 16 18:58:22.902841 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:58:22.902817    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f341c79_8d5a_4cad_8b28_a05e51f7aec9.slice/crio-d2492a63c0eb464e281cbdb97ac6614a9263b6597b1d36bb1e46595353eca898 WatchSource:0}: Error finding container d2492a63c0eb464e281cbdb97ac6614a9263b6597b1d36bb1e46595353eca898: Status 404 returned error can't find the container with id d2492a63c0eb464e281cbdb97ac6614a9263b6597b1d36bb1e46595353eca898
Sep 16 18:58:22.904103 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:22.904081    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:58:23.238222 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:23.238187    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh" event={"ID":"1f341c79-8d5a-4cad-8b28-a05e51f7aec9","Type":"ContainerStarted","Data":"d2492a63c0eb464e281cbdb97ac6614a9263b6597b1d36bb1e46595353eca898"}
Sep 16 18:58:24.244303 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:24.244262    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh" event={"ID":"1f341c79-8d5a-4cad-8b28-a05e51f7aec9","Type":"ContainerStarted","Data":"82663015e46ddddcbcc2c7864970a7bacf752a97473cf08fd8ebc11e6860c1ff"}
Sep 16 18:58:24.244707 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:24.244338    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:24.263807 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:24.263752    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh" podStartSLOduration=1.836218674 podStartE2EDuration="2.263735053s" totalImagesPullingTime="427.516379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:24.262686741 +0000 UTC m=+2527.821854603" watchObservedRunningTime="2026-09-16 18:58:24.263735053 +0000 UTC m=+2527.822902885"
Sep 16 18:58:29.606305 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.606260    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"]
Sep 16 18:58:29.606700 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.606499    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh" podUID="1f341c79-8d5a-4cad-8b28-a05e51f7aec9" containerName="maas-api" containerID="cri-o://82663015e46ddddcbcc2c7864970a7bacf752a97473cf08fd8ebc11e6860c1ff" gracePeriod=30
Sep 16 18:58:29.611877 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.611854    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:29.683532 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.683500    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"]
Sep 16 18:58:29.686990 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.686971    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"
Sep 16 18:58:29.709900 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.709871    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"]
Sep 16 18:58:29.739561 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.739530    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38707014-db1c-4d32-995f-abb332500ab8-metrics-certs\") pod \"maas-api-7589f5744d-wwk5s\" (UID: \"38707014-db1c-4d32-995f-abb332500ab8\") " pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"
Sep 16 18:58:29.739691 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.739573    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9c6t\" (UniqueName: \"kubernetes.io/projected/38707014-db1c-4d32-995f-abb332500ab8-kube-api-access-v9c6t\") pod \"maas-api-7589f5744d-wwk5s\" (UID: \"38707014-db1c-4d32-995f-abb332500ab8\") " pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"
Sep 16 18:58:29.739738 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.739704    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38707014-db1c-4d32-995f-abb332500ab8-openshift-service-ca\") pod \"maas-api-7589f5744d-wwk5s\" (UID: \"38707014-db1c-4d32-995f-abb332500ab8\") " pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"
Sep 16 18:58:29.739775 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.739755    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38707014-db1c-4d32-995f-abb332500ab8-maas-api-tls\") pod \"maas-api-7589f5744d-wwk5s\" (UID: \"38707014-db1c-4d32-995f-abb332500ab8\") " pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"
Sep 16 18:58:29.840660 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.840628    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38707014-db1c-4d32-995f-abb332500ab8-openshift-service-ca\") pod \"maas-api-7589f5744d-wwk5s\" (UID: \"38707014-db1c-4d32-995f-abb332500ab8\") " pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"
Sep 16 18:58:29.840795 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.840679    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38707014-db1c-4d32-995f-abb332500ab8-maas-api-tls\") pod \"maas-api-7589f5744d-wwk5s\" (UID: \"38707014-db1c-4d32-995f-abb332500ab8\") " pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"
Sep 16 18:58:29.840795 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.840745    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38707014-db1c-4d32-995f-abb332500ab8-metrics-certs\") pod \"maas-api-7589f5744d-wwk5s\" (UID: \"38707014-db1c-4d32-995f-abb332500ab8\") " pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"
Sep 16 18:58:29.840795 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.840767    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9c6t\" (UniqueName: \"kubernetes.io/projected/38707014-db1c-4d32-995f-abb332500ab8-kube-api-access-v9c6t\") pod \"maas-api-7589f5744d-wwk5s\" (UID: \"38707014-db1c-4d32-995f-abb332500ab8\") " pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"
Sep 16 18:58:29.841489 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.841454    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38707014-db1c-4d32-995f-abb332500ab8-openshift-service-ca\") pod \"maas-api-7589f5744d-wwk5s\" (UID: \"38707014-db1c-4d32-995f-abb332500ab8\") " pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"
Sep 16 18:58:29.843194 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.843171    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38707014-db1c-4d32-995f-abb332500ab8-metrics-certs\") pod \"maas-api-7589f5744d-wwk5s\" (UID: \"38707014-db1c-4d32-995f-abb332500ab8\") " pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"
Sep 16 18:58:29.843432 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.843413    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38707014-db1c-4d32-995f-abb332500ab8-maas-api-tls\") pod \"maas-api-7589f5744d-wwk5s\" (UID: \"38707014-db1c-4d32-995f-abb332500ab8\") " pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"
Sep 16 18:58:29.854395 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.854362    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9c6t\" (UniqueName: \"kubernetes.io/projected/38707014-db1c-4d32-995f-abb332500ab8-kube-api-access-v9c6t\") pod \"maas-api-7589f5744d-wwk5s\" (UID: \"38707014-db1c-4d32-995f-abb332500ab8\") " pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"
Sep 16 18:58:29.866726 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.866678    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:29.941791 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.941756    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfchg\" (UniqueName: \"kubernetes.io/projected/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-kube-api-access-bfchg\") pod \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\" (UID: \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\") "
Sep 16 18:58:29.941791 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.941809    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-metrics-certs\") pod \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\" (UID: \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\") "
Sep 16 18:58:29.942051 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.941870    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-maas-api-tls\") pod \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\" (UID: \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\") "
Sep 16 18:58:29.942051 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.941938    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-openshift-service-ca\") pod \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\" (UID: \"1f341c79-8d5a-4cad-8b28-a05e51f7aec9\") "
Sep 16 18:58:29.942542 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.942511    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f341c79-8d5a-4cad-8b28-a05e51f7aec9" (UID: "1f341c79-8d5a-4cad-8b28-a05e51f7aec9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:29.944128 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.944094    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-kube-api-access-bfchg" (OuterVolumeSpecName: "kube-api-access-bfchg") pod "1f341c79-8d5a-4cad-8b28-a05e51f7aec9" (UID: "1f341c79-8d5a-4cad-8b28-a05e51f7aec9"). InnerVolumeSpecName "kube-api-access-bfchg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:29.944431 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.944405    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f341c79-8d5a-4cad-8b28-a05e51f7aec9" (UID: "1f341c79-8d5a-4cad-8b28-a05e51f7aec9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:29.944520 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.944433    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1f341c79-8d5a-4cad-8b28-a05e51f7aec9" (UID: "1f341c79-8d5a-4cad-8b28-a05e51f7aec9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:29.998071 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:29.998040    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"
Sep 16 18:58:30.043196 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:30.043162    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-openshift-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:58:30.043196 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:30.043195    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfchg\" (UniqueName: \"kubernetes.io/projected/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-kube-api-access-bfchg\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:58:30.043447 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:30.043211    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-metrics-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:58:30.043447 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:30.043224    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f341c79-8d5a-4cad-8b28-a05e51f7aec9-maas-api-tls\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Sep 16 18:58:30.276661 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:30.276625    2554 generic.go:356] "Generic (PLEG): container finished" podID="1f341c79-8d5a-4cad-8b28-a05e51f7aec9" containerID="82663015e46ddddcbcc2c7864970a7bacf752a97473cf08fd8ebc11e6860c1ff" exitCode=0
Sep 16 18:58:30.276934 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:30.276669    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"
Sep 16 18:58:30.276934 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:30.276672    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh" event={"ID":"1f341c79-8d5a-4cad-8b28-a05e51f7aec9","Type":"ContainerDied","Data":"82663015e46ddddcbcc2c7864970a7bacf752a97473cf08fd8ebc11e6860c1ff"}
Sep 16 18:58:30.276934 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:30.276821    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh" event={"ID":"1f341c79-8d5a-4cad-8b28-a05e51f7aec9","Type":"ContainerDied","Data":"d2492a63c0eb464e281cbdb97ac6614a9263b6597b1d36bb1e46595353eca898"}
Sep 16 18:58:30.276934 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:30.276857    2554 scope.go:117] "RemoveContainer" containerID="82663015e46ddddcbcc2c7864970a7bacf752a97473cf08fd8ebc11e6860c1ff"
Sep 16 18:58:30.283422 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:30.283400    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"]
Sep 16 18:58:30.287143 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:30.287121    2554 scope.go:117] "RemoveContainer" containerID="82663015e46ddddcbcc2c7864970a7bacf752a97473cf08fd8ebc11e6860c1ff"
Sep 16 18:58:30.287409 ip-10-0-136-38 kubenswrapper[2554]: E0916 18:58:30.287389    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82663015e46ddddcbcc2c7864970a7bacf752a97473cf08fd8ebc11e6860c1ff\": container with ID starting with 82663015e46ddddcbcc2c7864970a7bacf752a97473cf08fd8ebc11e6860c1ff not found: ID does not exist" containerID="82663015e46ddddcbcc2c7864970a7bacf752a97473cf08fd8ebc11e6860c1ff"
Sep 16 18:58:30.287486 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:30.287415    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82663015e46ddddcbcc2c7864970a7bacf752a97473cf08fd8ebc11e6860c1ff"} err="failed to get container status \"82663015e46ddddcbcc2c7864970a7bacf752a97473cf08fd8ebc11e6860c1ff\": rpc error: code = NotFound desc = could not find container \"82663015e46ddddcbcc2c7864970a7bacf752a97473cf08fd8ebc11e6860c1ff\": container with ID starting with 82663015e46ddddcbcc2c7864970a7bacf752a97473cf08fd8ebc11e6860c1ff not found: ID does not exist"
Sep 16 18:58:30.288185 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:58:30.288167    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38707014_db1c_4d32_995f_abb332500ab8.slice/crio-58d7c1b572bcb30809af8071de4827ced4ddcc743dc12c0e6e329ffe7212a151 WatchSource:0}: Error finding container 58d7c1b572bcb30809af8071de4827ced4ddcc743dc12c0e6e329ffe7212a151: Status 404 returned error can't find the container with id 58d7c1b572bcb30809af8071de4827ced4ddcc743dc12c0e6e329ffe7212a151
Sep 16 18:58:30.309720 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:30.309699    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"]
Sep 16 18:58:30.313443 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:30.313424    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-b66c78b8-8nzvh"]
Sep 16 18:58:30.427487 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:30.427453    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:58:30.883503 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:30.883410    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f341c79-8d5a-4cad-8b28-a05e51f7aec9" path="/var/lib/kubelet/pods/1f341c79-8d5a-4cad-8b28-a05e51f7aec9/volumes"
Sep 16 18:58:31.284116 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:31.284085    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s" event={"ID":"38707014-db1c-4d32-995f-abb332500ab8","Type":"ContainerStarted","Data":"86d639e7aa7af1ad2f80dd8e1a305c887c762d603b180900d0f193b692810510"}
Sep 16 18:58:31.284335 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:31.284130    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s" event={"ID":"38707014-db1c-4d32-995f-abb332500ab8","Type":"ContainerStarted","Data":"58d7c1b572bcb30809af8071de4827ced4ddcc743dc12c0e6e329ffe7212a151"}
Sep 16 18:58:31.284335 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:31.284221    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"
Sep 16 18:58:31.309744 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:31.309694    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s" podStartSLOduration=1.891061826 podStartE2EDuration="2.309675098s" totalImagesPullingTime="418.613272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:31.308724129 +0000 UTC m=+2534.867891965" watchObservedRunningTime="2026-09-16 18:58:31.309675098 +0000 UTC m=+2534.868842928"
Sep 16 18:58:37.295804 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:37.295773    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7589f5744d-wwk5s"
Sep 16 18:58:38.308110 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:38.308075    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:58:44.123139 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:44.123102    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:58:51.035872 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:58:51.035833    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvdrk"]
Sep 16 18:59:00.358397 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:00.358358    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ls77m_6cc1f4df-c6ea-4a32-ba5e-d442c47e2665/manager/0.log"
Sep 16 18:59:00.484904 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:00.484875    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-q8q9l_adf3b276-9306-4c42-827d-e08851a774b6/manager/0.log"
Sep 16 18:59:00.810368 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:00.810338    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-rxrjz_d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978/server/0.log"
Sep 16 18:59:00.934282 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:00.934253    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mmq55_5c40361f-f58e-4143-80d6-c2a70e17acdb/manager/0.log"
Sep 16 18:59:01.332142 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:01.332113    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86bdbbf975-w4fdb_c77db88a-0d6c-4094-ad8e-f25cbefc7bc5/copy-manifests/0.log"
Sep 16 18:59:01.341410 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:01.341388    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86bdbbf975-w4fdb_c77db88a-0d6c-4094-ad8e-f25cbefc7bc5/manager/0.log"
Sep 16 18:59:01.808719 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:01.808688    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7589f5744d-wwk5s_38707014-db1c-4d32-995f-abb332500ab8/maas-api/0.log"
Sep 16 18:59:01.929646 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:01.929620    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826390-rsnjk_15fdbf2a-51fc-4498-abc1-92e00c38ebc9/cleanup/0.log"
Sep 16 18:59:02.188984 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:02.188915    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-ln86k_d6dfc071-3732-4c49-986b-b121a18f5f52/postgres/0.log"
Sep 16 18:59:03.277249 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:03.277199    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-vjxkp_0ec2d7cb-90f5-410b-8886-ce49c5accdc0/manager/0.log"
Sep 16 18:59:03.539354 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:03.539277    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-ks8vt_ce11585d-55c4-476c-8e3d-aad3e662ed4c/manager/0.log"
Sep 16 18:59:03.753391 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:03.753364    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-rvdrk_b2c4f1de-925d-4090-9ed7-9532521ac154/limitador/0.log"
Sep 16 18:59:05.803351 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:05.803316    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-rsldb_dffdb8a6-8524-4413-9fc2-5e4cffa61c80/main/0.log"
Sep 16 18:59:06.165133 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:06.165057    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-98mxv_2fd5a91b-c7c4-4fe0-b86e-54210860d8a2/main/0.log"
Sep 16 18:59:11.519473 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:11.519438    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-slrm4/must-gather-j8q9x"]
Sep 16 18:59:11.519843 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:11.519807    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f341c79-8d5a-4cad-8b28-a05e51f7aec9" containerName="maas-api"
Sep 16 18:59:11.519843 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:11.519818    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f341c79-8d5a-4cad-8b28-a05e51f7aec9" containerName="maas-api"
Sep 16 18:59:11.519917 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:11.519885    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f341c79-8d5a-4cad-8b28-a05e51f7aec9" containerName="maas-api"
Sep 16 18:59:11.523108 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:11.523091    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slrm4/must-gather-j8q9x"
Sep 16 18:59:11.527328 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:11.527305    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-slrm4\"/\"default-dockercfg-bzh68\""
Sep 16 18:59:11.527433 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:11.527305    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-slrm4\"/\"openshift-service-ca.crt\""
Sep 16 18:59:11.527433 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:11.527305    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-slrm4\"/\"kube-root-ca.crt\""
Sep 16 18:59:11.540576 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:11.540545    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-slrm4/must-gather-j8q9x"]
Sep 16 18:59:11.622138 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:11.622106    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/30a47dcb-6a61-46b8-be7a-82c23fc4f7b7-must-gather-output\") pod \"must-gather-j8q9x\" (UID: \"30a47dcb-6a61-46b8-be7a-82c23fc4f7b7\") " pod="openshift-must-gather-slrm4/must-gather-j8q9x"
Sep 16 18:59:11.622316 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:11.622142    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqtqd\" (UniqueName: \"kubernetes.io/projected/30a47dcb-6a61-46b8-be7a-82c23fc4f7b7-kube-api-access-qqtqd\") pod \"must-gather-j8q9x\" (UID: \"30a47dcb-6a61-46b8-be7a-82c23fc4f7b7\") " pod="openshift-must-gather-slrm4/must-gather-j8q9x"
Sep 16 18:59:11.723039 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:11.723005    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/30a47dcb-6a61-46b8-be7a-82c23fc4f7b7-must-gather-output\") pod \"must-gather-j8q9x\" (UID: \"30a47dcb-6a61-46b8-be7a-82c23fc4f7b7\") " pod="openshift-must-gather-slrm4/must-gather-j8q9x"
Sep 16 18:59:11.723039 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:11.723042    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqtqd\" (UniqueName: \"kubernetes.io/projected/30a47dcb-6a61-46b8-be7a-82c23fc4f7b7-kube-api-access-qqtqd\") pod \"must-gather-j8q9x\" (UID: \"30a47dcb-6a61-46b8-be7a-82c23fc4f7b7\") " pod="openshift-must-gather-slrm4/must-gather-j8q9x"
Sep 16 18:59:11.723380 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:11.723358    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/30a47dcb-6a61-46b8-be7a-82c23fc4f7b7-must-gather-output\") pod \"must-gather-j8q9x\" (UID: \"30a47dcb-6a61-46b8-be7a-82c23fc4f7b7\") " pod="openshift-must-gather-slrm4/must-gather-j8q9x"
Sep 16 18:59:11.736295 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:11.736270    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqtqd\" (UniqueName: \"kubernetes.io/projected/30a47dcb-6a61-46b8-be7a-82c23fc4f7b7-kube-api-access-qqtqd\") pod \"must-gather-j8q9x\" (UID: \"30a47dcb-6a61-46b8-be7a-82c23fc4f7b7\") " pod="openshift-must-gather-slrm4/must-gather-j8q9x"
Sep 16 18:59:11.831910 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:11.831814    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slrm4/must-gather-j8q9x"
Sep 16 18:59:12.140332 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:12.140308    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-slrm4/must-gather-j8q9x"]
Sep 16 18:59:12.142365 ip-10-0-136-38 kubenswrapper[2554]: W0916 18:59:12.142337    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30a47dcb_6a61_46b8_be7a_82c23fc4f7b7.slice/crio-f2c360b210104c75cd9808c38b36d615bc5f87bb4221ccf82b29db70a17102e1 WatchSource:0}: Error finding container f2c360b210104c75cd9808c38b36d615bc5f87bb4221ccf82b29db70a17102e1: Status 404 returned error can't find the container with id f2c360b210104c75cd9808c38b36d615bc5f87bb4221ccf82b29db70a17102e1
Sep 16 18:59:12.483478 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:12.483446    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slrm4/must-gather-j8q9x" event={"ID":"30a47dcb-6a61-46b8-be7a-82c23fc4f7b7","Type":"ContainerStarted","Data":"f2c360b210104c75cd9808c38b36d615bc5f87bb4221ccf82b29db70a17102e1"}
Sep 16 18:59:13.490143 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:13.490105    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slrm4/must-gather-j8q9x" event={"ID":"30a47dcb-6a61-46b8-be7a-82c23fc4f7b7","Type":"ContainerStarted","Data":"3ca1c39a87654b5d54f394f576340455b83d4662e0eacc42bde4db4c8d5e5f9a"}
Sep 16 18:59:13.490569 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:13.490156    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slrm4/must-gather-j8q9x" event={"ID":"30a47dcb-6a61-46b8-be7a-82c23fc4f7b7","Type":"ContainerStarted","Data":"be57e51fb2bf70482f207dc79965ba788127cf2b7cc994df051d226add904b03"}
Sep 16 18:59:13.520652 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:13.520597    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-slrm4/must-gather-j8q9x" podStartSLOduration=1.769453567 podStartE2EDuration="2.520579835s" totalImagesPullingTime="751.126268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:13.519531072 +0000 UTC m=+2577.078698907" watchObservedRunningTime="2026-09-16 18:59:13.520579835 +0000 UTC m=+2577.079747664"
Sep 16 18:59:15.282577 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:15.282543    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7w74g_50b4667a-4d35-44cb-865b-747a72c7c3e6/global-pull-secret-syncer/0.log"
Sep 16 18:59:15.624439 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:15.624367    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jtdvj_f06204d6-bd2b-45fe-b2bc-0814ad88beb7/konnectivity-agent/0.log"
Sep 16 18:59:15.920314 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:15.920206    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-38.ec2.internal_0f74f6b140c4aa2324e29421d89be3fe/haproxy/0.log"
Sep 16 18:59:17.319974 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:17.319928    2554 scope.go:117] "RemoveContainer" containerID="bb5453f21d78ee8dcdeb3e94f88942aa0b8bfb8102284ad14a7a56bc042473b6"
Sep 16 18:59:19.997530 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:19.997494    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-vjxkp_0ec2d7cb-90f5-410b-8886-ce49c5accdc0/manager/0.log"
Sep 16 18:59:20.143308 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:20.143277    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-ks8vt_ce11585d-55c4-476c-8e3d-aad3e662ed4c/manager/0.log"
Sep 16 18:59:20.212376 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:20.212338    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-rvdrk_b2c4f1de-925d-4090-9ed7-9532521ac154/limitador/0.log"
Sep 16 18:59:21.916878 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:21.916781    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_403782cd-2a03-492e-a394-f483a9db911d/alertmanager/0.log"
Sep 16 18:59:21.948356 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:21.948328    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_403782cd-2a03-492e-a394-f483a9db911d/config-reloader/0.log"
Sep 16 18:59:21.976323 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:21.976289    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_403782cd-2a03-492e-a394-f483a9db911d/kube-rbac-proxy-web/0.log"
Sep 16 18:59:22.005183 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:22.005139    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_403782cd-2a03-492e-a394-f483a9db911d/kube-rbac-proxy/0.log"
Sep 16 18:59:22.054004 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:22.053975    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_403782cd-2a03-492e-a394-f483a9db911d/kube-rbac-proxy-metric/0.log"
Sep 16 18:59:22.097936 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:22.097905    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_403782cd-2a03-492e-a394-f483a9db911d/prom-label-proxy/0.log"
Sep 16 18:59:22.132797 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:22.132758    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_403782cd-2a03-492e-a394-f483a9db911d/init-config-reloader/0.log"
Sep 16 18:59:22.677368 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:22.677259    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvf8s_7b64bf66-02e1-4598-8815-161511160ff4/node-exporter/0.log"
Sep 16 18:59:22.710264 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:22.710201    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvf8s_7b64bf66-02e1-4598-8815-161511160ff4/kube-rbac-proxy/0.log"
Sep 16 18:59:22.741244 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:22.741199    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvf8s_7b64bf66-02e1-4598-8815-161511160ff4/init-textfile/0.log"
Sep 16 18:59:22.907533 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:22.907483    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b7f4318-80cc-40ac-b06e-5250fa2a9730/prometheus/0.log"
Sep 16 18:59:22.969616 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:22.969536    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b7f4318-80cc-40ac-b06e-5250fa2a9730/config-reloader/0.log"
Sep 16 18:59:23.029372 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:23.029314    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b7f4318-80cc-40ac-b06e-5250fa2a9730/thanos-sidecar/0.log"
Sep 16 18:59:23.074270 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:23.074221    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b7f4318-80cc-40ac-b06e-5250fa2a9730/kube-rbac-proxy-web/0.log"
Sep 16 18:59:23.121935 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:23.121902    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b7f4318-80cc-40ac-b06e-5250fa2a9730/kube-rbac-proxy/0.log"
Sep 16 18:59:23.159107 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:23.159078    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b7f4318-80cc-40ac-b06e-5250fa2a9730/kube-rbac-proxy-thanos/0.log"
Sep 16 18:59:23.194780 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:23.194746    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7b7f4318-80cc-40ac-b06e-5250fa2a9730/init-config-reloader/0.log"
Sep 16 18:59:23.244417 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:23.244384    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-pd59k_0bf91448-f095-489f-813b-0a290a5ff26b/prometheus-operator/0.log"
Sep 16 18:59:23.294398 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:23.294369    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-pd59k_0bf91448-f095-489f-813b-0a290a5ff26b/kube-rbac-proxy/0.log"
Sep 16 18:59:23.342565 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:23.342529    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-l5xrt_bc98c7b9-3108-4ee2-b7c4-189338bcabc3/prometheus-operator-admission-webhook/0.log"
Sep 16 18:59:24.226848 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.226690    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"]
Sep 16 18:59:24.240665 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.240564    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.257574 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.257537    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"]
Sep 16 18:59:24.355747 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.355702    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5d72\" (UniqueName: \"kubernetes.io/projected/31986005-c060-45d3-af25-70d30af959cd-kube-api-access-p5d72\") pod \"perf-node-gather-daemonset-zgfcr\" (UID: \"31986005-c060-45d3-af25-70d30af959cd\") " pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.355952 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.355845    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31986005-c060-45d3-af25-70d30af959cd-lib-modules\") pod \"perf-node-gather-daemonset-zgfcr\" (UID: \"31986005-c060-45d3-af25-70d30af959cd\") " pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.355952 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.355912    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31986005-c060-45d3-af25-70d30af959cd-sys\") pod \"perf-node-gather-daemonset-zgfcr\" (UID: \"31986005-c060-45d3-af25-70d30af959cd\") " pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.356079 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.356007    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31986005-c060-45d3-af25-70d30af959cd-proc\") pod \"perf-node-gather-daemonset-zgfcr\" (UID: \"31986005-c060-45d3-af25-70d30af959cd\") " pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.356142 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.356082    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31986005-c060-45d3-af25-70d30af959cd-podres\") pod \"perf-node-gather-daemonset-zgfcr\" (UID: \"31986005-c060-45d3-af25-70d30af959cd\") " pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.457616 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.457580    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31986005-c060-45d3-af25-70d30af959cd-podres\") pod \"perf-node-gather-daemonset-zgfcr\" (UID: \"31986005-c060-45d3-af25-70d30af959cd\") " pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.457832 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.457650    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5d72\" (UniqueName: \"kubernetes.io/projected/31986005-c060-45d3-af25-70d30af959cd-kube-api-access-p5d72\") pod \"perf-node-gather-daemonset-zgfcr\" (UID: \"31986005-c060-45d3-af25-70d30af959cd\") " pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.457832 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.457703    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31986005-c060-45d3-af25-70d30af959cd-lib-modules\") pod \"perf-node-gather-daemonset-zgfcr\" (UID: \"31986005-c060-45d3-af25-70d30af959cd\") " pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.457832 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.457725    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31986005-c060-45d3-af25-70d30af959cd-sys\") pod \"perf-node-gather-daemonset-zgfcr\" (UID: \"31986005-c060-45d3-af25-70d30af959cd\") " pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.457832 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.457775    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31986005-c060-45d3-af25-70d30af959cd-proc\") pod \"perf-node-gather-daemonset-zgfcr\" (UID: \"31986005-c060-45d3-af25-70d30af959cd\") " pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.457832 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.457803    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31986005-c060-45d3-af25-70d30af959cd-podres\") pod \"perf-node-gather-daemonset-zgfcr\" (UID: \"31986005-c060-45d3-af25-70d30af959cd\") " pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.458119 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.457860    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31986005-c060-45d3-af25-70d30af959cd-proc\") pod \"perf-node-gather-daemonset-zgfcr\" (UID: \"31986005-c060-45d3-af25-70d30af959cd\") " pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.458119 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.457982    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31986005-c060-45d3-af25-70d30af959cd-sys\") pod \"perf-node-gather-daemonset-zgfcr\" (UID: \"31986005-c060-45d3-af25-70d30af959cd\") " pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.458119 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.458001    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31986005-c060-45d3-af25-70d30af959cd-lib-modules\") pod \"perf-node-gather-daemonset-zgfcr\" (UID: \"31986005-c060-45d3-af25-70d30af959cd\") " pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.469028 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.468997    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5d72\" (UniqueName: \"kubernetes.io/projected/31986005-c060-45d3-af25-70d30af959cd-kube-api-access-p5d72\") pod \"perf-node-gather-daemonset-zgfcr\" (UID: \"31986005-c060-45d3-af25-70d30af959cd\") " pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.554259 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.554186    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:24.838911 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:24.838814    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"]
Sep 16 18:59:25.578566 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:25.578531    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr" event={"ID":"31986005-c060-45d3-af25-70d30af959cd","Type":"ContainerStarted","Data":"43aa06e3c8b5b0356407c5179fc945a55e182e794bbd8c76fc40e39b7b2bd8ec"}
Sep 16 18:59:25.579050 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:25.578589    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr" event={"ID":"31986005-c060-45d3-af25-70d30af959cd","Type":"ContainerStarted","Data":"5a0a294c1f01094ceec9607b787abd7a315d7efa9fc637c6293ae8181180eb29"}
Sep 16 18:59:25.579050 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:25.578632    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:25.811111 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:25.811071    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/2.log"
Sep 16 18:59:25.815883 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:25.815859    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bz98l_aab214d7-cd3b-4018-b947-4e6c626eea04/console-operator/3.log"
Sep 16 18:59:28.635018 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:28.634976    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pbwcp_90a6aab5-3c38-4775-ba9c-3dc99fc92207/dns/0.log"
Sep 16 18:59:28.675709 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:28.675683    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pbwcp_90a6aab5-3c38-4775-ba9c-3dc99fc92207/kube-rbac-proxy/0.log"
Sep 16 18:59:28.831265 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:28.831213    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dqj4w_d52c2148-fdca-4bb1-9ef7-b2492c8b7b57/dns-node-resolver/0.log"
Sep 16 18:59:29.744374 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:29.744347    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b6hff_e1ee1605-7d05-4668-98e6-8249ac3e1052/node-ca/0.log"
Sep 16 18:59:31.594590 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:31.594559    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr"
Sep 16 18:59:31.623490 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:31.623422    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-slrm4/perf-node-gather-daemonset-zgfcr" podStartSLOduration=7.623401133 podStartE2EDuration="7.623401133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:25.604248534 +0000 UTC m=+2589.163416361" watchObservedRunningTime="2026-09-16 18:59:31.623401133 +0000 UTC m=+2595.182568966"
Sep 16 18:59:32.924075 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:32.924042    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zs8q4_9569d7bb-5ebe-4d49-8a92-faed5784b8c7/serve-healthcheck-canary/0.log"
Sep 16 18:59:33.684641 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:33.684563    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5z599_f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1/kube-rbac-proxy/0.log"
Sep 16 18:59:33.751020 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:33.750989    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5z599_f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1/exporter/0.log"
Sep 16 18:59:33.805962 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:33.805921    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5z599_f853fe08-8ee6-4e56-92fd-6a2e9cd7a6b1/extractor/0.log"
Sep 16 18:59:37.392630 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:37.392583    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ls77m_6cc1f4df-c6ea-4a32-ba5e-d442c47e2665/manager/0.log"
Sep 16 18:59:37.754681 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:37.754647    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-q8q9l_adf3b276-9306-4c42-827d-e08851a774b6/manager/0.log"
Sep 16 18:59:37.932436 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:37.932406    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-rxrjz_d41bbcd3-42ce-48d6-b7fa-fb51e4fdd978/server/0.log"
Sep 16 18:59:38.057321 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:38.057187    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mmq55_5c40361f-f58e-4143-80d6-c2a70e17acdb/manager/0.log"
Sep 16 18:59:38.374769 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:38.374705    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86bdbbf975-w4fdb_c77db88a-0d6c-4094-ad8e-f25cbefc7bc5/manager/0.log"
Sep 16 18:59:38.412460 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:38.412436    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86bdbbf975-w4fdb_c77db88a-0d6c-4094-ad8e-f25cbefc7bc5/copy-manifests/0.log"
Sep 16 18:59:39.756474 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:39.756351    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6fb494f6fb-hkmhl_63b5c5d7-959c-4ca1-ac14-10074310bcaf/manager/0.log"
Sep 16 18:59:39.892326 ip-10-0-136-38 kubenswrapper[2554]: I0916 18:59:39.892297    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-m4558_0d001f5b-8956-4d13-9be3-ba7abe8c65ff/openshift-lws-operator/0.log"