Sep 07 12:53:01.459516 ip-10-0-141-58 systemd[1]: Starting Kubernetes Kubelet...
Sep 07 12:53:01.968155 ip-10-0-141-58 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 12:53:01.968155 ip-10-0-141-58 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 07 12:53:01.968155 ip-10-0-141-58 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 12:53:01.968155 ip-10-0-141-58 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 07 12:53:01.968155 ip-10-0-141-58 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 12:53:01.970675 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.970585    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 07 12:53:01.974192 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974174    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 12:53:01.974239 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974194    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 12:53:01.974239 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974198    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 12:53:01.974239 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974201    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 12:53:01.974239 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974205    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 12:53:01.974239 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974220    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 12:53:01.974239 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974223    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 12:53:01.974239 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974226    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 12:53:01.974239 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974229    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 12:53:01.974239 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974233    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 12:53:01.974239 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974237    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 12:53:01.974239 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974241    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 12:53:01.974239 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974244    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 12:53:01.974239 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974246    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974249    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974252    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974255    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974258    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974260    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974263    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974265    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974268    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974271    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974274    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974276    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974279    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974282    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974285    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974287    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974290    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974292    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974295    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 12:53:01.974550 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974299    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974302    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974305    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974308    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974311    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974314    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974316    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974319    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974322    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974324    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974327    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974330    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974332    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974335    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974337    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974341    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974343    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974346    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974349    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974352    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 12:53:01.974987 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974354    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974357    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974359    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974362    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974366    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974368    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974371    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974374    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974377    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974380    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974383    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974386    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974388    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974391    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974393    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974396    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974399    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974401    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974404    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974407    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 12:53:01.975488 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974410    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974412    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974416    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974418    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974421    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974423    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974426    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974429    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974432    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974434    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974437    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974440    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974442    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974445    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974447    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974450    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974452    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974455    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974457    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974461    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 12:53:01.975966 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974463    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974467    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974471    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974474    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974477    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974480    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.974484    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975200    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975207    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975225    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975228    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975231    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975235    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975237    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975240    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975242    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975245    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975248    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 12:53:01.976454 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975251    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975255    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975258    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975260    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975263    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975265    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975268    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975271    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975273    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975276    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975279    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975282    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975344    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975350    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975354    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975359    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975361    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975364    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975367    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 12:53:01.976880 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975369    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975372    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975374    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975377    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975379    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975382    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975384    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975387    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975390    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975393    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975396    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975399    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975401    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975404    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975408    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975411    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975414    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975416    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975418    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 12:53:01.977360 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975421    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975424    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975426    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975429    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975431    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975434    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975436    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975439    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975441    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975444    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975447    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975450    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975452    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975455    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975457    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975460    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975462    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975464    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975468    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975472    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 12:53:01.977823 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975475    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975478    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975481    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975484    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975487    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975490    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975493    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975497    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975500    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975502    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975505    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975508    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975510    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975513    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975515    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975517    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975520    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975522    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975524    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975527    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 12:53:01.978361 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975531    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975535    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975538    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975540    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975543    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975545    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975548    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975550    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975552    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975555    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975953    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975957    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975971    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975974    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975977    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975981    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975983    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975986    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975988    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975992    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 12:53:01.978833 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975995    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.975998    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976000    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976003    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976005    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976008    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976010    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976013    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976016    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976018    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976021    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976024    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976026    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976029    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976031    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976034    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976036    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976039    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976041    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976044    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 12:53:01.979320 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976046    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976049    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976051    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976054    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976057    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976059    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976062    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976065    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976067    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976070    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976072    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976075    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976078    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976082    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976086    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976089    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976092    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976094    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976101    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976104    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 12:53:01.979841 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976106    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976109    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976111    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976114    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976116    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976119    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976122    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976124    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976127    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976131    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976133    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976136    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976138    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976141    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976144    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976147    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976149    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976152    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976154    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976157    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 12:53:01.980335 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976160    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976163    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976166    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976173    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976176    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976178    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976181    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976183    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976186    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976188    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976191    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976193    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976196    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976198    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976201    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976205    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976222    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976227    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976230    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 12:53:01.980815 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976234    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976237    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976239    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976242    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976245    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976248    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976250    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976253    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976256    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.976258    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976342    2565 flags.go:64] FLAG: --address="0.0.0.0"
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976378    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976885    2565 flags.go:64] FLAG: --anonymous-auth="true"
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976890    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976896    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976900    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976905    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976909    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976912    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976915    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976919    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 07 12:53:01.981326 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976922    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976925    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976928    2565 flags.go:64] FLAG: --cgroup-root=""
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976931    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976934    2565 flags.go:64] FLAG: --client-ca-file=""
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976937    2565 flags.go:64] FLAG: --cloud-provider="external"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976940    2565 flags.go:64] FLAG: --cluster-dns="[]"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976945    2565 flags.go:64] FLAG: --cluster-domain=""
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976947    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976951    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976954    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976957    2565 flags.go:64] FLAG: --container-log-max-files="5"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976961    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976964    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976967    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976971    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976974    2565 flags.go:64] FLAG: --contention-profiling="false"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976979    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976981    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976984    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976988    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976992    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976995    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.976997    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977000    2565 flags.go:64] FLAG: --enable-load-reader="false"
Sep 07 12:53:01.982244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977004    2565 flags.go:64] FLAG: --enable-server="true"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977008    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977013    2565 flags.go:64] FLAG: --event-burst="100"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977016    2565 flags.go:64] FLAG: --event-qps="50"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977019    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977023    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977026    2565 flags.go:64] FLAG: --eviction-hard=""
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977030    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977033    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977036    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977039    2565 flags.go:64] FLAG: --eviction-soft=""
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977042    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977045    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977048    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977050    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977053    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977056    2565 flags.go:64] FLAG: --fail-swap-on="true"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977059    2565 flags.go:64] FLAG: --feature-gates=""
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977063    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977066    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977069    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977072    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977075    2565 flags.go:64] FLAG: --healthz-port="10248"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977078    2565 flags.go:64] FLAG: --help="false"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977083    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-141-58.ec2.internal"
Sep 07 12:53:01.983258 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977086    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977089    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977092    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977096    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977099    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977102    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977105    2565 flags.go:64] FLAG: --image-service-endpoint=""
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977107    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977110    2565 flags.go:64] FLAG: --kube-api-burst="100"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977114    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977117    2565 flags.go:64] FLAG: --kube-api-qps="50"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977120    2565 flags.go:64] FLAG: --kube-reserved=""
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977123    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977126    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977128    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977131    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977134    2565 flags.go:64] FLAG: --lock-file=""
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977137    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977140    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977143    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977148    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977151    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977153    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 07 12:53:01.984269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977156    2565 flags.go:64] FLAG: --logging-format="text"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977159    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977162    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977165    2565 flags.go:64] FLAG: --manifest-url=""
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977168    2565 flags.go:64] FLAG: --manifest-url-header=""
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977172    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977175    2565 flags.go:64] FLAG: --max-open-files="1000000"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977179    2565 flags.go:64] FLAG: --max-pods="110"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977183    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977186    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977189    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977192    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977195    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977198    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977201    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977220    2565 flags.go:64] FLAG: --node-status-max-images="50"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977224    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977227    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977230    2565 flags.go:64] FLAG: --pod-cidr=""
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977234    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977239    2565 flags.go:64] FLAG: --pod-manifest-path=""
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977242    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977245    2565 flags.go:64] FLAG: --pods-per-core="0"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977248    2565 flags.go:64] FLAG: --port="10250"
Sep 07 12:53:01.984842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977251    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977254    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ef8678658cc05fd9"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977257    2565 flags.go:64] FLAG: --qos-reserved=""
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977259    2565 flags.go:64] FLAG: --read-only-port="10255"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977262    2565 flags.go:64] FLAG: --register-node="true"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977265    2565 flags.go:64] FLAG: --register-with-taints=""
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977269    2565 flags.go:64] FLAG: --registry-burst="10"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977272    2565 flags.go:64] FLAG: --registry-qps="5"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977274    2565 flags.go:64] FLAG: --reserved-cpus=""
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977277    2565 flags.go:64] FLAG: --reserved-memory=""
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977281    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977284    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977287    2565 flags.go:64] FLAG: --rotate-certificates="false"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977290    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977293    2565 flags.go:64] FLAG: --runonce="false"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977295    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977305    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977308    2565 flags.go:64] FLAG: --seccomp-default="false"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977311    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977315    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977318    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977321    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977324    2565 flags.go:64] FLAG: --storage-driver-password="root"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977327    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977330    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 07 12:53:01.985432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977333    2565 flags.go:64] FLAG: --storage-driver-user="root"
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977336    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977340    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977344    2565 flags.go:64] FLAG: --system-cgroups=""
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977347    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977352    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977355    2565 flags.go:64] FLAG: --tls-cert-file=""
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977358    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977363    2565 flags.go:64] FLAG: --tls-min-version=""
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977366    2565 flags.go:64] FLAG: --tls-private-key-file=""
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977368    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977371    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977374    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977377    2565 flags.go:64] FLAG: --v="2"
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977386    2565 flags.go:64] FLAG: --version="false"
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977389    2565 flags.go:64] FLAG: --vmodule=""
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977393    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.977396    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977517    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977522    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977525    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977528    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977531    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977535    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 12:53:01.986060 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977538    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977541    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977544    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977547    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977549    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977552    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977554    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977557    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977560    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977562    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977566    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977569    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977572    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977575    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977577    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977580    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977582    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977585    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977588    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977590    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 12:53:01.986665 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977593    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977595    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977598    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977601    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977604    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977606    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977609    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977611    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977614    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977616    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977619    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977622    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977625    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977627    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977630    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977632    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977635    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977637    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977640    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977643    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 12:53:01.987139 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977645    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977648    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977651    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977654    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977656    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977659    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977662    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977665    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977667    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977670    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977672    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977675    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977677    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977680    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977682    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977685    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977687    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977690    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977692    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 12:53:01.987647 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977695    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977698    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977702    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977705    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977709    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977712    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977715    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977717    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977720    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977724    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977727    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977730    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977733    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977736    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977738    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977744    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977747    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977750    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 12:53:01.988181 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977752    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 12:53:01.988650 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977756    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 12:53:01.988650 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977760    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 12:53:01.988650 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977763    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 12:53:01.988650 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977766    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 12:53:01.988650 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977769    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 12:53:01.988650 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977772    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 12:53:01.988650 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977774    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 12:53:01.988650 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977777    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 12:53:01.988650 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977780    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 12:53:01.988650 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977783    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 12:53:01.988650 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977786    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 12:53:01.988650 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977788    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 12:53:01.988650 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977791    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 12:53:01.988650 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977794    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 12:53:01.988650 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.977796    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 12:53:01.989022 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.978202    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 12:53:01.989022 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.985901    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 07 12:53:01.989022 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.985918    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 07 12:53:01.989022 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.985938    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 07 12:53:01.989022 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.985945    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 07 12:53:01.989022 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986012    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 12:53:01.989022 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986017    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 12:53:01.989022 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986021    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 12:53:01.989022 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986024    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 12:53:01.989022 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986026    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 12:53:01.989022 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986030    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 12:53:01.989022 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986032    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 12:53:01.989022 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986035    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 12:53:01.989022 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986037    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 12:53:01.989022 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986040    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986043    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986045    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986048    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986050    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986053    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986056    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986060    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986064    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986067    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986070    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986073    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986076    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986079    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986081    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986084    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986086    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986089    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986091    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 12:53:01.989446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986094    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986098    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986101    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986104    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986107    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986109    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986112    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986114    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986117    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986119    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986122    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986124    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986127    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986129    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986132    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986134    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986137    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986139    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986142    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986144    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 12:53:01.989891 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986148    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986151    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986154    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986157    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986168    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986172    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986174    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986177    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986180    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986185    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986188    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986190    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986193    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986197    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986202    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986205    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986223    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986227    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986230    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 12:53:01.990383 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986233    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986235    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986238    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986241    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986243    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986246    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986249    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986251    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986254    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986256    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986259    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986262    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986264    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986267    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986270    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986272    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986275    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986277    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986280    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 12:53:01.990839 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986282    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 12:53:01.991309 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986285    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 12:53:01.991309 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986287    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 12:53:01.991309 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986290    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 12:53:01.991309 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986292    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 12:53:01.991309 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986295    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 12:53:01.991309 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986298    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 12:53:01.991309 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986301    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 12:53:01.991309 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986304    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 12:53:01.991309 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986307    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 12:53:01.991309 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986310    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 12:53:01.991309 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986313    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 12:53:01.991309 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986315    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 12:53:01.991309 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.986320    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 12:53:01.991309 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986460    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 12:53:01.991309 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986466    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986470    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986473    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986477    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986481    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986485    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986488    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986491    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986493    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986496    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986499    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986501    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986504    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986507    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986509    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986512    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986514    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986516    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986519    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 12:53:01.991674 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986521    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986524    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986526    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986529    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986531    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986534    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986537    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986540    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986543    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986546    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986548    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986551    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986554    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986556    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986559    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986561    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986564    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986566    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986569    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986571    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 12:53:01.992143 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986574    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986576    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986579    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986581    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986583    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986586    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986588    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986591    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986593    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986596    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986598    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986600    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986603    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986605    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986608    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986610    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986613    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986616    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986618    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986621    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 12:53:01.992723 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986623    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986630    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986633    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986636    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986638    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986641    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986643    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986646    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986648    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986651    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986653    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986655    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986658    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986660    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986663    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986665    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986668    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986670    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986673    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 12:53:01.993293 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986677    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986681    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986683    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986686    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986688    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986691    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986694    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986696    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986699    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986701    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986704    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986707    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986709    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986712    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986714    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986718    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986720    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986723    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986725    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 12:53:01.993769 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:01.986728    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 12:53:01.994491 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.986732    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 12:53:01.994491 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.986842    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 07 12:53:01.994491 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.988677    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 07 12:53:01.994491 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.989586    2565 server.go:1019] "Starting client certificate rotation"
Sep 07 12:53:01.994491 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.989685    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 12:53:01.994491 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.989729    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 12:53:01.994491 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:01.991896    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 12:53:02.000947 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.000925    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 12:53:02.013251 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.013227    2565 log.go:25] "Validated CRI v1 runtime API"
Sep 07 12:53:02.015952 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.015939    2565 log.go:25] "Validated CRI v1 image API"
Sep 07 12:53:02.016794 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.016770    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 07 12:53:02.021418 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.021387    2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b42b6734-344c-41ca-b4d9-e59dc44bc632:/dev/nvme0n1p3 bb4de181-d728-4031-977c-29352060a303:/dev/nvme0n1p4]
Sep 07 12:53:02.021490 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.021418    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 07 12:53:02.024391 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.024375    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 12:53:02.026944 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.026840    2565 manager.go:216] Machine: {Timestamp:2026-09-07 12:53:02.025549414 +0000 UTC m=+0.348530664 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2499996 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec277aef0ca69c87b4e11165cd983575 SystemUUID:ec277aef-0ca6-9c87-b4e1-1165cd983575 BootID:ceb0ab61-16db-44b7-bb4e-c1b37d436838 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:06:d5:eb:3a:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:06:d5:eb:3a:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:20:69:a7:0b:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 07 12:53:02.026944 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.026940    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 07 12:53:02.027102 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.027089    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 07 12:53:02.029894 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.029864    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 07 12:53:02.030032 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.029898    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-58.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 07 12:53:02.030658 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.030646    2565 topology_manager.go:138] "Creating topology manager with none policy"
Sep 07 12:53:02.030696 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.030661    2565 container_manager_linux.go:309] "Creating device plugin manager"
Sep 07 12:53:02.030696 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.030669    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 12:53:02.031376 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.031365    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 12:53:02.032106 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.032097    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 07 12:53:02.035551 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.035539    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 12:53:02.035660 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.035651    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 07 12:53:02.035692 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.035681    2565 kubelet.go:486] "Attempting to sync node with API server"
Sep 07 12:53:02.035692 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.035692    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 07 12:53:02.035771 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.035711    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 07 12:53:02.035771 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.035721    2565 kubelet.go:398] "Adding apiserver pod source"
Sep 07 12:53:02.035771 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.035730    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 07 12:53:02.036116 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.036105    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 12:53:02.036166 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.036123    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 12:53:02.039066 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.039042    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 07 12:53:02.041156 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.041142    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 07 12:53:02.041205 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.041173    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 07 12:53:02.042183 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.042172    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 07 12:53:02.042231 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.042188    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 07 12:53:02.042231 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.042194    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 07 12:53:02.042231 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.042199    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 07 12:53:02.042231 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.042225    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 07 12:53:02.042231 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.042231    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 07 12:53:02.042374 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.042245    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 07 12:53:02.042374 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.042252    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 07 12:53:02.042374 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.042259    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 07 12:53:02.042374 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.042264    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 07 12:53:02.042374 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.042272    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 07 12:53:02.042374 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.042277    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 07 12:53:02.043094 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.043086    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 07 12:53:02.043094 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.043094    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 07 12:53:02.047120 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.046988    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 07 12:53:02.047237 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.047166    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-58.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 07 12:53:02.047315 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.047270    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-58.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 07 12:53:02.047826 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.047802    2565 server.go:1268] "Started kubelet"
Sep 07 12:53:02.047991 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.047958    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 07 12:53:02.048083 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.047955    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 07 12:53:02.048173 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.048106    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 07 12:53:02.048360 ip-10-0-141-58 systemd[1]: Started Kubernetes Kubelet.
Sep 07 12:53:02.049145 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.049121    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 07 12:53:02.049781 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.049762    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6fwpv"
Sep 07 12:53:02.050303 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.050290    2565 server.go:310] "Adding debug handlers to kubelet server"
Sep 07 12:53:02.053326 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.053302    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 07 12:53:02.054045 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.052967    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-58.ec2.internal.18d30b026d4c2a24  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-58.ec2.internal,UID:ip-10-0-141-58.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-58.ec2.internal,},FirstTimestamp:2026-09-07 12:53:02.047767076 +0000 UTC m=+0.370748327,LastTimestamp:2026-09-07 12:53:02.047767076 +0000 UTC m=+0.370748327,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-58.ec2.internal,}"
Sep 07 12:53:02.055161 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.055147    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 07 12:53:02.055797 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.055782    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 07 12:53:02.056043 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.056025    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found"
Sep 07 12:53:02.056395 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.056377    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 07 12:53:02.056483 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.056401    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 07 12:53:02.056483 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.056377    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 07 12:53:02.056483 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.056464    2565 reconstruct.go:97] "Volume reconstruction finished"
Sep 07 12:53:02.056483 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.056470    2565 reconciler.go:29] "Reconciler: start to sync state"
Sep 07 12:53:02.056635 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.056512    2565 factory.go:55] Registering systemd factory
Sep 07 12:53:02.056635 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.056542    2565 factory.go:223] Registration of the systemd container factory successfully
Sep 07 12:53:02.056785 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.056744    2565 factory.go:153] Registering CRI-O factory
Sep 07 12:53:02.056785 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.056759    2565 factory.go:223] Registration of the crio container factory successfully
Sep 07 12:53:02.056785 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.056802    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 07 12:53:02.056785 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.056814    2565 factory.go:103] Registering Raw factory
Sep 07 12:53:02.056785 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.056825    2565 manager.go:1196] Started watching for new ooms in manager
Sep 07 12:53:02.057174 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.057161    2565 manager.go:349] Starting recovery of all containers
Sep 07 12:53:02.059131 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.059107    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6fwpv"
Sep 07 12:53:02.066081 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.066062    2565 manager.go:354] Recovery completed
Sep 07 12:53:02.067058 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.067028    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-58.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 07 12:53:02.067954 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.067907    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 07 12:53:02.073619 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.073605    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 12:53:02.075468 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.075451    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 12:53:02.075542 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.075480    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 12:53:02.075542 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.075490    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientPID"
Sep 07 12:53:02.076037 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.076021    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 07 12:53:02.076037 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.076037    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 07 12:53:02.076120 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.076053    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 12:53:02.076120 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.076085    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 12:53:02.076796 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.076781    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 12:53:02.076853 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.076808    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 12:53:02.076853 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.076823    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientPID"
Sep 07 12:53:02.078267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.078255    2565 policy_none.go:49] "None policy: Start"
Sep 07 12:53:02.078319 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.078274    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 07 12:53:02.078319 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.078289    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 07 12:53:02.080294 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.080280    2565 policy_none.go:47] "Start"
Sep 07 12:53:02.137385 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.117767    2565 manager.go:337] "Starting Device Plugin manager"
Sep 07 12:53:02.137385 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.117802    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 07 12:53:02.137385 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.117812    2565 server.go:87] "Starting device plugin registration server"
Sep 07 12:53:02.137385 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.118066    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 07 12:53:02.137385 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.118080    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 07 12:53:02.137385 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.118297    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 07 12:53:02.137385 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.118388    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 07 12:53:02.137385 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.118397    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 07 12:53:02.137385 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.118836    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 07 12:53:02.137385 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.118898    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-58.ec2.internal\" not found"
Sep 07 12:53:02.187878 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.187839    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 07 12:53:02.189180 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.189158    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 07 12:53:02.189277 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.189226    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 07 12:53:02.189277 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.189258    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 07 12:53:02.189382 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.189306    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 07 12:53:02.191817 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.191799    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 12:53:02.218976 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.218904    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 12:53:02.219858 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.219839    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 12:53:02.219954 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.219870    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 12:53:02.219954 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.219881    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientPID"
Sep 07 12:53:02.219954 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.219904    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.226387 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.226371    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.226461 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.226394    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-58.ec2.internal\": node \"ip-10-0-141-58.ec2.internal\" not found"
Sep 07 12:53:02.244109 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.244090    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found"
Sep 07 12:53:02.289691 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.289652    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal"]
Sep 07 12:53:02.289815 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.289740    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 12:53:02.291337 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.291321    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 12:53:02.291427 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.291352    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 12:53:02.291427 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.291366    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientPID"
Sep 07 12:53:02.293732 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.293719    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 12:53:02.293855 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.293842    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.293893 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.293873    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 12:53:02.294486 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.294469    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 12:53:02.294550 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.294496    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 12:53:02.294550 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.294506    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientPID"
Sep 07 12:53:02.294550 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.294521    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 12:53:02.294550 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.294537    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 12:53:02.294550 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.294546    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientPID"
Sep 07 12:53:02.296721 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.296708    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.296782 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.296731    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 12:53:02.297524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.297510    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 12:53:02.297588 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.297532    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 12:53:02.297588 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.297540    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientPID"
Sep 07 12:53:02.319888 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.319862    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-58.ec2.internal\" not found" node="ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.324163 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.324149    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-58.ec2.internal\" not found" node="ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.345118 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.345095    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found"
Sep 07 12:53:02.357411 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.357380    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/80dcc41252cdc8837afdd88dc326d069-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal\" (UID: \"80dcc41252cdc8837afdd88dc326d069\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.357512 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.357417    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80dcc41252cdc8837afdd88dc326d069-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal\" (UID: \"80dcc41252cdc8837afdd88dc326d069\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.357512 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.357441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/899f0bbad87a70907b72c03ee3b7d029-config\") pod \"kube-apiserver-proxy-ip-10-0-141-58.ec2.internal\" (UID: \"899f0bbad87a70907b72c03ee3b7d029\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.446180 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.446139    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found"
Sep 07 12:53:02.458532 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.458506    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/80dcc41252cdc8837afdd88dc326d069-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal\" (UID: \"80dcc41252cdc8837afdd88dc326d069\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.458605 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.458537    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/80dcc41252cdc8837afdd88dc326d069-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal\" (UID: \"80dcc41252cdc8837afdd88dc326d069\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.458605 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.458583    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80dcc41252cdc8837afdd88dc326d069-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal\" (UID: \"80dcc41252cdc8837afdd88dc326d069\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.458605 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.458602    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/899f0bbad87a70907b72c03ee3b7d029-config\") pod \"kube-apiserver-proxy-ip-10-0-141-58.ec2.internal\" (UID: \"899f0bbad87a70907b72c03ee3b7d029\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.458700 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.458626    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/899f0bbad87a70907b72c03ee3b7d029-config\") pod \"kube-apiserver-proxy-ip-10-0-141-58.ec2.internal\" (UID: \"899f0bbad87a70907b72c03ee3b7d029\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.458700 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.458651    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80dcc41252cdc8837afdd88dc326d069-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal\" (UID: \"80dcc41252cdc8837afdd88dc326d069\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.546984 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.546910    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found"
Sep 07 12:53:02.622396 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.622364    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.627325 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.627227    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal"
Sep 07 12:53:02.647975 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.647747    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found"
Sep 07 12:53:02.748305 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.748263    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found"
Sep 07 12:53:02.848831 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.848745    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found"
Sep 07 12:53:02.949391 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:02.949348    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found"
Sep 07 12:53:02.989824 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.989795    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 07 12:53:02.990285 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:02.989950    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 12:53:03.049894 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:03.049858    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found"
Sep 07 12:53:03.067123 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:03.067081    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 12:48:03 +0000 UTC" deadline="2028-05-28 09:01:24.100803216 +0000 UTC"
Sep 07 12:53:03.067123 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:03.067115    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15092h8m21.033691696s"
Sep 07 12:53:03.150345 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:03.150316    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found"
Sep 07 12:53:03.251303 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:03.251277    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found"
Sep 07 12:53:03.258998 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:03.258949    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899f0bbad87a70907b72c03ee3b7d029.slice/crio-e5323b57f9b1786d1ef1e520debc53d2ad6ed16401d34c2ae7e295e94d0e2d82 WatchSource:0}: Error finding container e5323b57f9b1786d1ef1e520debc53d2ad6ed16401d34c2ae7e295e94d0e2d82: Status 404 returned error can't find the container with id e5323b57f9b1786d1ef1e520debc53d2ad6ed16401d34c2ae7e295e94d0e2d82
Sep 07 12:53:03.259180 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:03.259165    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80dcc41252cdc8837afdd88dc326d069.slice/crio-c31a5dd78d57db448c5cf59e3b762b418e5076f717ee3a38aa61f08e08292ce8 WatchSource:0}: Error finding container c31a5dd78d57db448c5cf59e3b762b418e5076f717ee3a38aa61f08e08292ce8: Status 404 returned error can't find the container with id c31a5dd78d57db448c5cf59e3b762b418e5076f717ee3a38aa61f08e08292ce8
Sep 07 12:53:03.263664 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:03.263648    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 12:53:03.281392 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:03.281371    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 12:53:03.351903 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:03.351867    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found"
Sep 07 12:53:03.385281 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:03.385199    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 12:53:03.417918 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:03.417889    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 12:53:03.456114 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:03.456086    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal"
Sep 07 12:53:03.468509 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:03.468487    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 12:53:03.469434 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:03.469422    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal"
Sep 07 12:53:03.478156 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:03.478140    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 12:53:05.328666 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.328636    2565 apiserver.go:52] "Watching apiserver"
Sep 07 12:53:05.336225 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.336199    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 07 12:53:05.336679 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.336661    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-vjgrh","kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr","openshift-cluster-node-tuning-operator/tuned-c458f","openshift-dns/node-resolver-dqbpb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal","openshift-multus/network-metrics-daemon-cbgw4","openshift-network-diagnostics/network-check-target-5tbdn","kube-system/konnectivity-agent-w7hq9","openshift-image-registry/node-ca-7gz9f","openshift-multus/multus-additional-cni-plugins-f5wl5","openshift-multus/multus-sjshd","openshift-network-operator/iptables-alerter-9jfkw","openshift-ovn-kubernetes/ovnkube-node-smjnx"]
Sep 07 12:53:05.339312 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.339288    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:05.339439 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:05.339387    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:05.341629 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.341601    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9jfkw"
Sep 07 12:53:05.343960 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.343917    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.344791 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.344562    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4t7d6\""
Sep 07 12:53:05.344791 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.344585    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 07 12:53:05.344791 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.344609    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 07 12:53:05.344791 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.344729    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 07 12:53:05.346143 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.346121    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.346217 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.346152    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 07 12:53:05.346567 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.346544    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 07 12:53:05.346787 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.346770    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 07 12:53:05.346942 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.346770    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x4x5k\""
Sep 07 12:53:05.347874 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.347855    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 07 12:53:05.348512 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.348487    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 07 12:53:05.348512 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.348506    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xrfs8\""
Sep 07 12:53:05.348651 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.348552    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 07 12:53:05.349369 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.349269    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dqbpb"
Sep 07 12:53:05.351624 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.351427    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 07 12:53:05.351624 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.351460    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 07 12:53:05.351624 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.351595    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wzwnr\""
Sep 07 12:53:05.351829 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.351812    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.353900 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.353732    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 07 12:53:05.354420 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.354058    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 07 12:53:05.354420 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.354115    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w7hq9"
Sep 07 12:53:05.354420 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.354305    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 07 12:53:05.354598 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.354567    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 07 12:53:05.354598 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.354588    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 07 12:53:05.354698 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.354625    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 07 12:53:05.354698 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.354590    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wr69z\""
Sep 07 12:53:05.356281 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.356263    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 07 12:53:05.356800 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.356579    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:05.356935 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.356600    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-95wsq\""
Sep 07 12:53:05.356935 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:05.356856    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:05.356935 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.356677    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 07 12:53:05.362395 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.361673    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:05.362395 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:05.361762    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:05.362395 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.361902    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-systemd\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.362395 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.361937    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-tuned\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.362395 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.361963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs\") pod \"network-metrics-daemon-cbgw4\" (UID: \"4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4\") " pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:05.362395 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.361980    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpxjb\" (UniqueName: \"kubernetes.io/projected/93cdd73f-1603-4e5a-bde9-e5e967b168cd-kube-api-access-xpxjb\") pod \"iptables-alerter-9jfkw\" (UID: \"93cdd73f-1603-4e5a-bde9-e5e967b168cd\") " pod="openshift-network-operator/iptables-alerter-9jfkw"
Sep 07 12:53:05.362395 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.362395 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362026    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-device-dir\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.362395 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362046    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.362395 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362072    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-sys\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.362395 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qn4q\" (UniqueName: \"kubernetes.io/projected/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-kube-api-access-2qn4q\") pod \"network-metrics-daemon-cbgw4\" (UID: \"4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4\") " pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:05.362395 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362131    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93cdd73f-1603-4e5a-bde9-e5e967b168cd-host-slash\") pod \"iptables-alerter-9jfkw\" (UID: \"93cdd73f-1603-4e5a-bde9-e5e967b168cd\") " pod="openshift-network-operator/iptables-alerter-9jfkw"
Sep 07 12:53:05.362395 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-sysctl-d\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.362395 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362188    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-registration-dir\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.362395 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362216    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-sys-fs\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.362395 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362237    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-sysconfig\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.363237 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362268    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-lib-modules\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.363237 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362307    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93cdd73f-1603-4e5a-bde9-e5e967b168cd-iptables-alerter-script\") pod \"iptables-alerter-9jfkw\" (UID: \"93cdd73f-1603-4e5a-bde9-e5e967b168cd\") " pod="openshift-network-operator/iptables-alerter-9jfkw"
Sep 07 12:53:05.363237 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362353    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-socket-dir\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.363237 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362391    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-run\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.363237 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362418    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-var-lib-kubelet\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.363237 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362438    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-tmp\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.363237 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362462    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-modprobe-d\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.363237 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd2sq\" (UniqueName: \"kubernetes.io/projected/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-kube-api-access-vd2sq\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.363237 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362523    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaac4f3c-dfa1-4c7f-8e67-2e01a910c117-tmp-dir\") pod \"node-resolver-dqbpb\" (UID: \"aaac4f3c-dfa1-4c7f-8e67-2e01a910c117\") " pod="openshift-dns/node-resolver-dqbpb"
Sep 07 12:53:05.363237 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362548    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nq4j\" (UniqueName: \"kubernetes.io/projected/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-kube-api-access-8nq4j\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.363237 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362579    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-kubernetes\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.363237 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362608    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-sysctl-conf\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.363237 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362636    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-host\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.363237 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362664    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aaac4f3c-dfa1-4c7f-8e67-2e01a910c117-hosts-file\") pod \"node-resolver-dqbpb\" (UID: \"aaac4f3c-dfa1-4c7f-8e67-2e01a910c117\") " pod="openshift-dns/node-resolver-dqbpb"
Sep 07 12:53:05.363237 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.362690    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr5q9\" (UniqueName: \"kubernetes.io/projected/aaac4f3c-dfa1-4c7f-8e67-2e01a910c117-kube-api-access-pr5q9\") pod \"node-resolver-dqbpb\" (UID: \"aaac4f3c-dfa1-4c7f-8e67-2e01a910c117\") " pod="openshift-dns/node-resolver-dqbpb"
Sep 07 12:53:05.363945 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.363320    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7gz9f"
Sep 07 12:53:05.363945 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.363840    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 12:53:05.368353 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.364649    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.368353 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.365879    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 07 12:53:05.368353 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.365921    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 07 12:53:05.368353 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.366175    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8qtfh\""
Sep 07 12:53:05.368353 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.366196    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 07 12:53:05.368353 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.366719    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 07 12:53:05.368353 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.367028    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fjwcw\""
Sep 07 12:53:05.368353 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.367118    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 07 12:53:05.368353 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.367125    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 07 12:53:05.368353 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.367157    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 07 12:53:05.368353 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.367434    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.368353 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.367794    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 07 12:53:05.372192 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.372170    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 07 12:53:05.372286 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.372197    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sqq26\""
Sep 07 12:53:05.382046 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.382024    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mfc6w"
Sep 07 12:53:05.390387 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.390367    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mfc6w"
Sep 07 12:53:05.405012 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.404984    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 12:53:05.449753 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.449727    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 07 12:53:05.463789 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.463715    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-run-netns\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.463789 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.463760    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-node-log\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.463789 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.463787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.464025 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.463814    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c743a37-043a-4a2a-8d5b-dc151cc29e58-env-overrides\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.464025 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.463836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4450013-6c87-4e72-8722-20add36bb64a-host\") pod \"node-ca-7gz9f\" (UID: \"d4450013-6c87-4e72-8722-20add36bb64a\") " pod="openshift-image-registry/node-ca-7gz9f"
Sep 07 12:53:05.464025 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.463863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4c45d9f-f285-4600-b036-836ef644d234-cni-binary-copy\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.464025 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.463889    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4c45d9f-f285-4600-b036-836ef644d234-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.464025 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.463913    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-system-cni-dir\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.464025 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.463942    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nq4j\" (UniqueName: \"kubernetes.io/projected/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-kube-api-access-8nq4j\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.464025 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.463991    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-kubernetes\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.464025 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-sysctl-conf\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aaac4f3c-dfa1-4c7f-8e67-2e01a910c117-hosts-file\") pod \"node-resolver-dqbpb\" (UID: \"aaac4f3c-dfa1-4c7f-8e67-2e01a910c117\") " pod="openshift-dns/node-resolver-dqbpb"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464056    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-kubernetes\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464058    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr5q9\" (UniqueName: \"kubernetes.io/projected/aaac4f3c-dfa1-4c7f-8e67-2e01a910c117-kube-api-access-pr5q9\") pod \"node-resolver-dqbpb\" (UID: \"aaac4f3c-dfa1-4c7f-8e67-2e01a910c117\") " pod="openshift-dns/node-resolver-dqbpb"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464099    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-run-ovn\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464124    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-run-ovn-kubernetes\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-cni-netd\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464171    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret\") pod \"global-pull-secret-syncer-vjgrh\" (UID: \"b22d72c4-fb0e-4f78-9e7e-0550e6928616\") " pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464194    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-systemd-units\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464229    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn8w2\" (UniqueName: \"kubernetes.io/projected/6c743a37-043a-4a2a-8d5b-dc151cc29e58-kube-api-access-tn8w2\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464255    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59z24\" (UniqueName: \"kubernetes.io/projected/d4450013-6c87-4e72-8722-20add36bb64a-kube-api-access-59z24\") pod \"node-ca-7gz9f\" (UID: \"d4450013-6c87-4e72-8722-20add36bb64a\") " pod="openshift-image-registry/node-ca-7gz9f"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20fb9fad-025c-42bb-92e5-86b868e484b7-cni-binary-copy\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-tuned\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464348    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs\") pod \"network-metrics-daemon-cbgw4\" (UID: \"4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4\") " pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464372    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464394    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-sysctl-conf\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-device-dir\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.464425 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464398    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aaac4f3c-dfa1-4c7f-8e67-2e01a910c117-hosts-file\") pod \"node-resolver-dqbpb\" (UID: \"aaac4f3c-dfa1-4c7f-8e67-2e01a910c117\") " pod="openshift-dns/node-resolver-dqbpb"
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464464    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-device-dir\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464484    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-run-systemd\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-var-lib-kubelet\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464539    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qn4q\" (UniqueName: \"kubernetes.io/projected/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-kube-api-access-2qn4q\") pod \"network-metrics-daemon-cbgw4\" (UID: \"4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4\") " pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464552    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464574    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93cdd73f-1603-4e5a-bde9-e5e967b168cd-host-slash\") pod \"iptables-alerter-9jfkw\" (UID: \"93cdd73f-1603-4e5a-bde9-e5e967b168cd\") " pod="openshift-network-operator/iptables-alerter-9jfkw"
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464600    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-sysctl-d\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4c45d9f-f285-4600-b036-836ef644d234-cnibin\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:05.464632    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464650    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-var-lib-cni-multus\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464676    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-registration-dir\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:05.464719    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs podName:4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:05.964691479 +0000 UTC m=+2.995557770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs") pod "network-metrics-daemon-cbgw4" (UID: "4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464723    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464730    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-registration-dir\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464764    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-sys-fs\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.465176 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464816    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93cdd73f-1603-4e5a-bde9-e5e967b168cd-host-slash\") pod \"iptables-alerter-9jfkw\" (UID: \"93cdd73f-1603-4e5a-bde9-e5e967b168cd\") " pod="openshift-network-operator/iptables-alerter-9jfkw"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464878    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464987    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-sysctl-d\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.464732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-sys-fs\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-sysconfig\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465115    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4c45d9f-f285-4600-b036-836ef644d234-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-run-multus-certs\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465149    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93cdd73f-1603-4e5a-bde9-e5e967b168cd-iptables-alerter-script\") pod \"iptables-alerter-9jfkw\" (UID: \"93cdd73f-1603-4e5a-bde9-e5e967b168cd\") " pod="openshift-network-operator/iptables-alerter-9jfkw"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-run\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465179    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4c45d9f-f285-4600-b036-836ef644d234-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465195    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-var-lib-kubelet\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-modprobe-d\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465224    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-cni-bin\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465239    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-multus-conf-dir\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465255    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-run-netns\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465269    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zvmx\" (UniqueName: \"kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx\") pod \"network-check-target-5tbdn\" (UID: \"fbbea119-b216-4988-ba0d-7c408dac7fc6\") " pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465282    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b22d72c4-fb0e-4f78-9e7e-0550e6928616-dbus\") pod \"global-pull-secret-syncer-vjgrh\" (UID: \"b22d72c4-fb0e-4f78-9e7e-0550e6928616\") " pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:05.466072 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465295    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da9073e8-463b-4ec5-b1e2-03a9260d2471-agent-certs\") pod \"konnectivity-agent-w7hq9\" (UID: \"da9073e8-463b-4ec5-b1e2-03a9260d2471\") " pod="kube-system/konnectivity-agent-w7hq9"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-host\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465348    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-kubelet\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465365    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-log-socket\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465378    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da9073e8-463b-4ec5-b1e2-03a9260d2471-konnectivity-ca\") pod \"konnectivity-agent-w7hq9\" (UID: \"da9073e8-463b-4ec5-b1e2-03a9260d2471\") " pod="kube-system/konnectivity-agent-w7hq9"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-var-lib-cni-bin\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-systemd\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b22d72c4-fb0e-4f78-9e7e-0550e6928616-kubelet-config\") pod \"global-pull-secret-syncer-vjgrh\" (UID: \"b22d72c4-fb0e-4f78-9e7e-0550e6928616\") " pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465455    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4c45d9f-f285-4600-b036-836ef644d234-system-cni-dir\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465469    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-multus-cni-dir\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465482    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m46ms\" (UniqueName: \"kubernetes.io/projected/20fb9fad-025c-42bb-92e5-86b868e484b7-kube-api-access-m46ms\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465497    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c743a37-043a-4a2a-8d5b-dc151cc29e58-ovnkube-config\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465514    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpxjb\" (UniqueName: \"kubernetes.io/projected/93cdd73f-1603-4e5a-bde9-e5e967b168cd-kube-api-access-xpxjb\") pod \"iptables-alerter-9jfkw\" (UID: \"93cdd73f-1603-4e5a-bde9-e5e967b168cd\") " pod="openshift-network-operator/iptables-alerter-9jfkw"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-sys\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465547    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d4450013-6c87-4e72-8722-20add36bb64a-serviceca\") pod \"node-ca-7gz9f\" (UID: \"d4450013-6c87-4e72-8722-20add36bb64a\") " pod="openshift-image-registry/node-ca-7gz9f"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-multus-socket-dir-parent\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465576    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c743a37-043a-4a2a-8d5b-dc151cc29e58-ovnkube-script-lib\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.466828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465591    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-run-k8s-cni-cncf-io\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465608    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-hostroot\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-lib-modules\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465638    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-slash\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465642    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-var-lib-kubelet\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-etc-openvswitch\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465668    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c743a37-043a-4a2a-8d5b-dc151cc29e58-ovn-node-metrics-cert\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465681    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-cnibin\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465697    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20fb9fad-025c-42bb-92e5-86b868e484b7-multus-daemon-config\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-etc-kubernetes\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465718    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-run\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-socket-dir\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-tmp\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465769    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd2sq\" (UniqueName: \"kubernetes.io/projected/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-kube-api-access-vd2sq\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465787    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaac4f3c-dfa1-4c7f-8e67-2e01a910c117-tmp-dir\") pod \"node-resolver-dqbpb\" (UID: \"aaac4f3c-dfa1-4c7f-8e67-2e01a910c117\") " pod="openshift-dns/node-resolver-dqbpb"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465804    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-var-lib-openvswitch\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465824    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-run-openvswitch\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.467774 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk7g7\" (UniqueName: \"kubernetes.io/projected/a4c45d9f-f285-4600-b036-836ef644d234-kube-api-access-dk7g7\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.468577 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.465495    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-sysconfig\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.468577 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.466010    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-host\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.468577 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.466069    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-systemd\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.468577 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.466091    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93cdd73f-1603-4e5a-bde9-e5e967b168cd-iptables-alerter-script\") pod \"iptables-alerter-9jfkw\" (UID: \"93cdd73f-1603-4e5a-bde9-e5e967b168cd\") " pod="openshift-network-operator/iptables-alerter-9jfkw"
Sep 07 12:53:05.468577 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.466126    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-sys\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.468577 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.466206    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-modprobe-d\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.468577 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.466294    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-lib-modules\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.468577 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.466366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-socket-dir\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.468577 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.466432    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaac4f3c-dfa1-4c7f-8e67-2e01a910c117-tmp-dir\") pod \"node-resolver-dqbpb\" (UID: \"aaac4f3c-dfa1-4c7f-8e67-2e01a910c117\") " pod="openshift-dns/node-resolver-dqbpb"
Sep 07 12:53:05.469881 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.469858    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-tmp\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.474379 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.474355    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qn4q\" (UniqueName: \"kubernetes.io/projected/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-kube-api-access-2qn4q\") pod \"network-metrics-daemon-cbgw4\" (UID: \"4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4\") " pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:05.474474 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.474443    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr5q9\" (UniqueName: \"kubernetes.io/projected/aaac4f3c-dfa1-4c7f-8e67-2e01a910c117-kube-api-access-pr5q9\") pod \"node-resolver-dqbpb\" (UID: \"aaac4f3c-dfa1-4c7f-8e67-2e01a910c117\") " pod="openshift-dns/node-resolver-dqbpb"
Sep 07 12:53:05.474908 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.474888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nq4j\" (UniqueName: \"kubernetes.io/projected/efb6fcfc-8f7a-4157-a56e-6cbd63d931bd-kube-api-access-8nq4j\") pod \"aws-ebs-csi-driver-node-9z9gr\" (UID: \"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.475265 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.475244    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpxjb\" (UniqueName: \"kubernetes.io/projected/93cdd73f-1603-4e5a-bde9-e5e967b168cd-kube-api-access-xpxjb\") pod \"iptables-alerter-9jfkw\" (UID: \"93cdd73f-1603-4e5a-bde9-e5e967b168cd\") " pod="openshift-network-operator/iptables-alerter-9jfkw"
Sep 07 12:53:05.476647 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.476626    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd2sq\" (UniqueName: \"kubernetes.io/projected/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-kube-api-access-vd2sq\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.476726 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.476672    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0a3de58-4067-45e4-b1b4-7cfce37b0fe4-etc-tuned\") pod \"tuned-c458f\" (UID: \"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4\") " pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.486144 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.486095    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal" event={"ID":"899f0bbad87a70907b72c03ee3b7d029","Type":"ContainerStarted","Data":"e5323b57f9b1786d1ef1e520debc53d2ad6ed16401d34c2ae7e295e94d0e2d82"}
Sep 07 12:53:05.487194 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.487173    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" event={"ID":"80dcc41252cdc8837afdd88dc326d069","Type":"ContainerStarted","Data":"c31a5dd78d57db448c5cf59e3b762b418e5076f717ee3a38aa61f08e08292ce8"}
Sep 07 12:53:05.566817 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.566783    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c743a37-043a-4a2a-8d5b-dc151cc29e58-ovnkube-script-lib\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.566817 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.566824    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-run-k8s-cni-cncf-io\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.567060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.566849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-hostroot\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.567060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.566873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-slash\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.567060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.566896    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-etc-openvswitch\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.567060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.566904    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-run-k8s-cni-cncf-io\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.567060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.566958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-slash\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.567060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.566968    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-hostroot\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.567060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.566998    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-etc-openvswitch\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.567060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567022    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c743a37-043a-4a2a-8d5b-dc151cc29e58-ovn-node-metrics-cert\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.567060 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567046    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-cnibin\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567101    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-cnibin\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567160    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20fb9fad-025c-42bb-92e5-86b868e484b7-multus-daemon-config\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567195    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-etc-kubernetes\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567229    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-var-lib-openvswitch\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-run-openvswitch\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk7g7\" (UniqueName: \"kubernetes.io/projected/a4c45d9f-f285-4600-b036-836ef644d234-kube-api-access-dk7g7\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-etc-kubernetes\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567304    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-run-netns\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-node-log\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567363    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567376    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-var-lib-openvswitch\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c743a37-043a-4a2a-8d5b-dc151cc29e58-env-overrides\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4450013-6c87-4e72-8722-20add36bb64a-host\") pod \"node-ca-7gz9f\" (UID: \"d4450013-6c87-4e72-8722-20add36bb64a\") " pod="openshift-image-registry/node-ca-7gz9f"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-run-netns\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567431    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4c45d9f-f285-4600-b036-836ef644d234-cni-binary-copy\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567453    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4c45d9f-f285-4600-b036-836ef644d234-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.567524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567476    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-system-cni-dir\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-run-ovn\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567507    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-node-log\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-run-openvswitch\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567527    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-run-ovn-kubernetes\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c743a37-043a-4a2a-8d5b-dc151cc29e58-ovnkube-script-lib\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567687    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-system-cni-dir\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567735    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4450013-6c87-4e72-8722-20add36bb64a-host\") pod \"node-ca-7gz9f\" (UID: \"d4450013-6c87-4e72-8722-20add36bb64a\") " pod="openshift-image-registry/node-ca-7gz9f"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567760    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-cni-netd\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567772    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/20fb9fad-025c-42bb-92e5-86b868e484b7-multus-daemon-config\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret\") pod \"global-pull-secret-syncer-vjgrh\" (UID: \"b22d72c4-fb0e-4f78-9e7e-0550e6928616\") " pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567812    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-systemd-units\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn8w2\" (UniqueName: \"kubernetes.io/projected/6c743a37-043a-4a2a-8d5b-dc151cc29e58-kube-api-access-tn8w2\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-run-ovn\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567859    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59z24\" (UniqueName: \"kubernetes.io/projected/d4450013-6c87-4e72-8722-20add36bb64a-kube-api-access-59z24\") pod \"node-ca-7gz9f\" (UID: \"d4450013-6c87-4e72-8722-20add36bb64a\") " pod="openshift-image-registry/node-ca-7gz9f"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567882    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20fb9fad-025c-42bb-92e5-86b868e484b7-cni-binary-copy\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:05.567914    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 12:53:05.568052 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-run-systemd\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.568593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-var-lib-kubelet\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.568593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.567947    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4c45d9f-f285-4600-b036-836ef644d234-cni-binary-copy\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.568593 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:05.567977    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret podName:b22d72c4-fb0e-4f78-9e7e-0550e6928616 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:06.067959798 +0000 UTC m=+3.098826088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret") pod "global-pull-secret-syncer-vjgrh" (UID: "b22d72c4-fb0e-4f78-9e7e-0550e6928616") : object "kube-system"/"original-pull-secret" not registered
Sep 07 12:53:05.568593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568012    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-cni-netd\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.568593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568067    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c743a37-043a-4a2a-8d5b-dc151cc29e58-env-overrides\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.568593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568079    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4c45d9f-f285-4600-b036-836ef644d234-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.568593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-run-ovn-kubernetes\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.568593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4c45d9f-f285-4600-b036-836ef644d234-cnibin\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.568593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568158    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-run-systemd\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.568593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568168    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-systemd-units\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.568593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-var-lib-cni-multus\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.568593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568207    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4c45d9f-f285-4600-b036-836ef644d234-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.568593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568218    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-var-lib-kubelet\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.568593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568210    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4c45d9f-f285-4600-b036-836ef644d234-cnibin\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.568593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568234    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-run-multus-certs\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.568593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568266    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-run-multus-certs\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4c45d9f-f285-4600-b036-836ef644d234-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-cni-bin\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568311    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-var-lib-cni-multus\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568348    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-multus-conf-dir\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568371    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-cni-bin\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568373    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-run-netns\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568402    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvmx\" (UniqueName: \"kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx\") pod \"network-check-target-5tbdn\" (UID: \"fbbea119-b216-4988-ba0d-7c408dac7fc6\") " pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568423    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20fb9fad-025c-42bb-92e5-86b868e484b7-cni-binary-copy\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-multus-conf-dir\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568424    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b22d72c4-fb0e-4f78-9e7e-0550e6928616-dbus\") pod \"global-pull-secret-syncer-vjgrh\" (UID: \"b22d72c4-fb0e-4f78-9e7e-0550e6928616\") " pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568471    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b22d72c4-fb0e-4f78-9e7e-0550e6928616-dbus\") pod \"global-pull-secret-syncer-vjgrh\" (UID: \"b22d72c4-fb0e-4f78-9e7e-0550e6928616\") " pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568472    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da9073e8-463b-4ec5-b1e2-03a9260d2471-agent-certs\") pod \"konnectivity-agent-w7hq9\" (UID: \"da9073e8-463b-4ec5-b1e2-03a9260d2471\") " pod="kube-system/konnectivity-agent-w7hq9"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568507    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-kubelet\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568531    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-log-socket\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da9073e8-463b-4ec5-b1e2-03a9260d2471-konnectivity-ca\") pod \"konnectivity-agent-w7hq9\" (UID: \"da9073e8-463b-4ec5-b1e2-03a9260d2471\") " pod="kube-system/konnectivity-agent-w7hq9"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568579    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-var-lib-cni-bin\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568599    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-log-socket\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.569267 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568602    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b22d72c4-fb0e-4f78-9e7e-0550e6928616-kubelet-config\") pod \"global-pull-secret-syncer-vjgrh\" (UID: \"b22d72c4-fb0e-4f78-9e7e-0550e6928616\") " pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568403    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-run-netns\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4c45d9f-f285-4600-b036-836ef644d234-system-cni-dir\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568648    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-multus-cni-dir\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568655    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b22d72c4-fb0e-4f78-9e7e-0550e6928616-kubelet-config\") pod \"global-pull-secret-syncer-vjgrh\" (UID: \"b22d72c4-fb0e-4f78-9e7e-0550e6928616\") " pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568662    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46ms\" (UniqueName: \"kubernetes.io/projected/20fb9fad-025c-42bb-92e5-86b868e484b7-kube-api-access-m46ms\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568677    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c743a37-043a-4a2a-8d5b-dc151cc29e58-ovnkube-config\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568680    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4c45d9f-f285-4600-b036-836ef644d234-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568690    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-host-var-lib-cni-bin\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568598    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c743a37-043a-4a2a-8d5b-dc151cc29e58-host-kubelet\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568695    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d4450013-6c87-4e72-8722-20add36bb64a-serviceca\") pod \"node-ca-7gz9f\" (UID: \"d4450013-6c87-4e72-8722-20add36bb64a\") " pod="openshift-image-registry/node-ca-7gz9f"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-multus-socket-dir-parent\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568750    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4c45d9f-f285-4600-b036-836ef644d234-system-cni-dir\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568804    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-multus-cni-dir\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.568802    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/20fb9fad-025c-42bb-92e5-86b868e484b7-multus-socket-dir-parent\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.569039    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4c45d9f-f285-4600-b036-836ef644d234-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.569133    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da9073e8-463b-4ec5-b1e2-03a9260d2471-konnectivity-ca\") pod \"konnectivity-agent-w7hq9\" (UID: \"da9073e8-463b-4ec5-b1e2-03a9260d2471\") " pod="kube-system/konnectivity-agent-w7hq9"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.569257    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d4450013-6c87-4e72-8722-20add36bb64a-serviceca\") pod \"node-ca-7gz9f\" (UID: \"d4450013-6c87-4e72-8722-20add36bb64a\") " pod="openshift-image-registry/node-ca-7gz9f"
Sep 07 12:53:05.570049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.569394    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c743a37-043a-4a2a-8d5b-dc151cc29e58-ovnkube-config\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.570955 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.569782    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c743a37-043a-4a2a-8d5b-dc151cc29e58-ovn-node-metrics-cert\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.571016 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.571003    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da9073e8-463b-4ec5-b1e2-03a9260d2471-agent-certs\") pod \"konnectivity-agent-w7hq9\" (UID: \"da9073e8-463b-4ec5-b1e2-03a9260d2471\") " pod="kube-system/konnectivity-agent-w7hq9"
Sep 07 12:53:05.580280 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.580254    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk7g7\" (UniqueName: \"kubernetes.io/projected/a4c45d9f-f285-4600-b036-836ef644d234-kube-api-access-dk7g7\") pod \"multus-additional-cni-plugins-f5wl5\" (UID: \"a4c45d9f-f285-4600-b036-836ef644d234\") " pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.580445 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:05.580302    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 12:53:05.580445 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:05.580320    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 12:53:05.580445 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:05.580350    2565 projected.go:196] Error preparing data for projected volume kube-api-access-5zvmx for pod openshift-network-diagnostics/network-check-target-5tbdn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:53:05.580445 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:05.580405    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx podName:fbbea119-b216-4988-ba0d-7c408dac7fc6 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:06.080391231 +0000 UTC m=+3.111257542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5zvmx" (UniqueName: "kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx") pod "network-check-target-5tbdn" (UID: "fbbea119-b216-4988-ba0d-7c408dac7fc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:53:05.582747 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.582721    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn8w2\" (UniqueName: \"kubernetes.io/projected/6c743a37-043a-4a2a-8d5b-dc151cc29e58-kube-api-access-tn8w2\") pod \"ovnkube-node-smjnx\" (UID: \"6c743a37-043a-4a2a-8d5b-dc151cc29e58\") " pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.582918 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.582896    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59z24\" (UniqueName: \"kubernetes.io/projected/d4450013-6c87-4e72-8722-20add36bb64a-kube-api-access-59z24\") pod \"node-ca-7gz9f\" (UID: \"d4450013-6c87-4e72-8722-20add36bb64a\") " pod="openshift-image-registry/node-ca-7gz9f"
Sep 07 12:53:05.583034 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.583019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46ms\" (UniqueName: \"kubernetes.io/projected/20fb9fad-025c-42bb-92e5-86b868e484b7-kube-api-access-m46ms\") pod \"multus-sjshd\" (UID: \"20fb9fad-025c-42bb-92e5-86b868e484b7\") " pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.657255 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.657217    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9jfkw"
Sep 07 12:53:05.676753 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.676724    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c458f"
Sep 07 12:53:05.685505 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.685481    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr"
Sep 07 12:53:05.691249 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.691223    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dqbpb"
Sep 07 12:53:05.706194 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.706171    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:05.714136 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.714075    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w7hq9"
Sep 07 12:53:05.721905 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.721883    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7gz9f"
Sep 07 12:53:05.728626 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.728605    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f5wl5"
Sep 07 12:53:05.733314 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.733291    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sjshd"
Sep 07 12:53:05.971392 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:05.971302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs\") pod \"network-metrics-daemon-cbgw4\" (UID: \"4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4\") " pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:05.971553 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:05.971422    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:53:05.971553 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:05.971474    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs podName:4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:06.971460307 +0000 UTC m=+4.002326595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs") pod "network-metrics-daemon-cbgw4" (UID: "4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:53:06.071850 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:06.071820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret\") pod \"global-pull-secret-syncer-vjgrh\" (UID: \"b22d72c4-fb0e-4f78-9e7e-0550e6928616\") " pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:06.072020 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:06.071936    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 12:53:06.072020 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:06.072010    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret podName:b22d72c4-fb0e-4f78-9e7e-0550e6928616 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:07.071990949 +0000 UTC m=+4.102857260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret") pod "global-pull-secret-syncer-vjgrh" (UID: "b22d72c4-fb0e-4f78-9e7e-0550e6928616") : object "kube-system"/"original-pull-secret" not registered
Sep 07 12:53:06.172773 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:06.172736    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvmx\" (UniqueName: \"kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx\") pod \"network-check-target-5tbdn\" (UID: \"fbbea119-b216-4988-ba0d-7c408dac7fc6\") " pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:06.172924 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:06.172871    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 12:53:06.172924 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:06.172892    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 12:53:06.172924 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:06.172903    2565 projected.go:196] Error preparing data for projected volume kube-api-access-5zvmx for pod openshift-network-diagnostics/network-check-target-5tbdn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:53:06.173029 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:06.172964    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx podName:fbbea119-b216-4988-ba0d-7c408dac7fc6 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:07.172944025 +0000 UTC m=+4.203810326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zvmx" (UniqueName: "kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx") pod "network-check-target-5tbdn" (UID: "fbbea119-b216-4988-ba0d-7c408dac7fc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:53:06.393474 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:06.393397    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 12:48:05 +0000 UTC" deadline="2028-03-11 18:05:08.635406814 +0000 UTC"
Sep 07 12:53:06.393474 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:06.393429    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13229h12m2.241982896s"
Sep 07 12:53:06.428889 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:06.428853    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cdd73f_1603_4e5a_bde9_e5e967b168cd.slice/crio-5354e6fc2902abfe99cec32202b9a5e17c477e8f6fe769da59db3a35e2913262 WatchSource:0}: Error finding container 5354e6fc2902abfe99cec32202b9a5e17c477e8f6fe769da59db3a35e2913262: Status 404 returned error can't find the container with id 5354e6fc2902abfe99cec32202b9a5e17c477e8f6fe769da59db3a35e2913262
Sep 07 12:53:06.435211 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:06.435188    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb6fcfc_8f7a_4157_a56e_6cbd63d931bd.slice/crio-0128f4da1a742f943a2c4eca1669514328b244c80c7d4abff358e57f5abd0f07 WatchSource:0}: Error finding container 0128f4da1a742f943a2c4eca1669514328b244c80c7d4abff358e57f5abd0f07: Status 404 returned error can't find the container with id 0128f4da1a742f943a2c4eca1669514328b244c80c7d4abff358e57f5abd0f07
Sep 07 12:53:06.457631 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:06.457533    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9073e8_463b_4ec5_b1e2_03a9260d2471.slice/crio-83bb3ce67ea4c810d59a0753ce5e12738c704797bc41679cca167753c90ecec5 WatchSource:0}: Error finding container 83bb3ce67ea4c810d59a0753ce5e12738c704797bc41679cca167753c90ecec5: Status 404 returned error can't find the container with id 83bb3ce67ea4c810d59a0753ce5e12738c704797bc41679cca167753c90ecec5
Sep 07 12:53:06.458264 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:06.458243    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a3de58_4067_45e4_b1b4_7cfce37b0fe4.slice/crio-1375f58d0073d6e5e4181909959c386a373d2b20bb96d1b42ed810cf84e4e5b8 WatchSource:0}: Error finding container 1375f58d0073d6e5e4181909959c386a373d2b20bb96d1b42ed810cf84e4e5b8: Status 404 returned error can't find the container with id 1375f58d0073d6e5e4181909959c386a373d2b20bb96d1b42ed810cf84e4e5b8
Sep 07 12:53:06.461303 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:06.459718    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fb9fad_025c_42bb_92e5_86b868e484b7.slice/crio-caa79f17e628c665d4c2550fbc387fbdc15ee2484b12763286d4c33c4de4f022 WatchSource:0}: Error finding container caa79f17e628c665d4c2550fbc387fbdc15ee2484b12763286d4c33c4de4f022: Status 404 returned error can't find the container with id caa79f17e628c665d4c2550fbc387fbdc15ee2484b12763286d4c33c4de4f022
Sep 07 12:53:06.461593 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:06.461573    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaac4f3c_dfa1_4c7f_8e67_2e01a910c117.slice/crio-46a9254880a5aa15c1982303d863d336dfc760a708535f268c67e7f9014dea35 WatchSource:0}: Error finding container 46a9254880a5aa15c1982303d863d336dfc760a708535f268c67e7f9014dea35: Status 404 returned error can't find the container with id 46a9254880a5aa15c1982303d863d336dfc760a708535f268c67e7f9014dea35
Sep 07 12:53:06.482288 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:06.482258    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:06.482406 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:06.482264    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:06.482406 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:06.482387    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:06.482502 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:06.482449    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:06.494184 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:06.494159    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f5wl5" event={"ID":"a4c45d9f-f285-4600-b036-836ef644d234","Type":"ContainerStarted","Data":"35f0e76a5f23f1d6a72e4fc0dc8b2fb36de6162264430f2a4f29979e4a172ec7"}
Sep 07 12:53:06.496231 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:06.496209    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sjshd" event={"ID":"20fb9fad-025c-42bb-92e5-86b868e484b7","Type":"ContainerStarted","Data":"caa79f17e628c665d4c2550fbc387fbdc15ee2484b12763286d4c33c4de4f022"}
Sep 07 12:53:06.497429 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:06.497410    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dqbpb" event={"ID":"aaac4f3c-dfa1-4c7f-8e67-2e01a910c117","Type":"ContainerStarted","Data":"46a9254880a5aa15c1982303d863d336dfc760a708535f268c67e7f9014dea35"}
Sep 07 12:53:06.498771 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:06.498747    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7gz9f" event={"ID":"d4450013-6c87-4e72-8722-20add36bb64a","Type":"ContainerStarted","Data":"8a6d6d6dc51e157fbc4d05177020c01553af09f58037c02903f470acdf28bd61"}
Sep 07 12:53:06.499917 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:06.499894    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx" event={"ID":"6c743a37-043a-4a2a-8d5b-dc151cc29e58","Type":"ContainerStarted","Data":"7950bda0f31aa03bf58c878351a50ae092bd430e63ad935885cc8db4bec7354b"}
Sep 07 12:53:06.501095 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:06.501076    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9jfkw" event={"ID":"93cdd73f-1603-4e5a-bde9-e5e967b168cd","Type":"ContainerStarted","Data":"5354e6fc2902abfe99cec32202b9a5e17c477e8f6fe769da59db3a35e2913262"}
Sep 07 12:53:06.504179 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:06.504153    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c458f" event={"ID":"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4","Type":"ContainerStarted","Data":"1375f58d0073d6e5e4181909959c386a373d2b20bb96d1b42ed810cf84e4e5b8"}
Sep 07 12:53:06.506870 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:06.506819    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w7hq9" event={"ID":"da9073e8-463b-4ec5-b1e2-03a9260d2471","Type":"ContainerStarted","Data":"83bb3ce67ea4c810d59a0753ce5e12738c704797bc41679cca167753c90ecec5"}
Sep 07 12:53:06.508222 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:06.508201    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr" event={"ID":"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd","Type":"ContainerStarted","Data":"0128f4da1a742f943a2c4eca1669514328b244c80c7d4abff358e57f5abd0f07"}
Sep 07 12:53:06.981836 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:06.981612    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs\") pod \"network-metrics-daemon-cbgw4\" (UID: \"4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4\") " pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:06.981993 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:06.981888    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:53:06.981993 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:06.981971    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs podName:4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:08.981947664 +0000 UTC m=+6.012813968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs") pod "network-metrics-daemon-cbgw4" (UID: "4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:53:07.082639 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:07.082587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret\") pod \"global-pull-secret-syncer-vjgrh\" (UID: \"b22d72c4-fb0e-4f78-9e7e-0550e6928616\") " pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:07.082891 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:07.082872    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 12:53:07.082976 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:07.082958    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret podName:b22d72c4-fb0e-4f78-9e7e-0550e6928616 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:09.082935891 +0000 UTC m=+6.113802191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret") pod "global-pull-secret-syncer-vjgrh" (UID: "b22d72c4-fb0e-4f78-9e7e-0550e6928616") : object "kube-system"/"original-pull-secret" not registered
Sep 07 12:53:07.183786 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:07.183708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvmx\" (UniqueName: \"kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx\") pod \"network-check-target-5tbdn\" (UID: \"fbbea119-b216-4988-ba0d-7c408dac7fc6\") " pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:07.183895 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:07.183838    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 12:53:07.183895 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:07.183853    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 12:53:07.183895 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:07.183863    2565 projected.go:196] Error preparing data for projected volume kube-api-access-5zvmx for pod openshift-network-diagnostics/network-check-target-5tbdn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:53:07.187679 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:07.186995    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx podName:fbbea119-b216-4988-ba0d-7c408dac7fc6 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:09.18389493 +0000 UTC m=+6.214761221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zvmx" (UniqueName: "kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx") pod "network-check-target-5tbdn" (UID: "fbbea119-b216-4988-ba0d-7c408dac7fc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:53:07.402253 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:07.393976    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 12:48:05 +0000 UTC" deadline="2028-04-27 06:10:54.662942099 +0000 UTC"
Sep 07 12:53:07.402253 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:07.394040    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14345h17m47.268905653s"
Sep 07 12:53:07.484210 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:07.484166    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:07.484390 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:07.484296    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:07.515745 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:07.515701    2565 generic.go:356] "Generic (PLEG): container finished" podID="80dcc41252cdc8837afdd88dc326d069" containerID="d314b12564107abdd0423f27674302d89da8cf632df3202335d70b5d5f05324d" exitCode=0
Sep 07 12:53:07.515910 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:07.515801    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" event={"ID":"80dcc41252cdc8837afdd88dc326d069","Type":"ContainerDied","Data":"d314b12564107abdd0423f27674302d89da8cf632df3202335d70b5d5f05324d"}
Sep 07 12:53:07.521080 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:07.521027    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal" event={"ID":"899f0bbad87a70907b72c03ee3b7d029","Type":"ContainerStarted","Data":"54f08f88e628126e59a1695cabe19804054a3f06d9b9c47d956176945603ce9e"}
Sep 07 12:53:08.482231 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:08.482192    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:08.482727 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:08.482348    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:08.482727 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:08.482368    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:08.482727 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:08.482545    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:08.526423 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:08.526387    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" event={"ID":"80dcc41252cdc8837afdd88dc326d069","Type":"ContainerStarted","Data":"4ccf11f390bf416afda7f7ef09f4745a210a781e38c7358094a9fdebf77edcb4"}
Sep 07 12:53:08.544224 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:08.543854    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal" podStartSLOduration=4.543833369 podStartE2EDuration="4.543833369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:53:07.550069346 +0000 UTC m=+4.580935657" watchObservedRunningTime="2026-09-07 12:53:08.543833369 +0000 UTC m=+5.574699681"
Sep 07 12:53:08.544224 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:08.544117    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" podStartSLOduration=4.54411095 podStartE2EDuration="4.54411095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:53:08.543490256 +0000 UTC m=+5.574356567" watchObservedRunningTime="2026-09-07 12:53:08.54411095 +0000 UTC m=+5.574977260"
Sep 07 12:53:08.999027 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:08.998986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs\") pod \"network-metrics-daemon-cbgw4\" (UID: \"4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4\") " pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:08.999209 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:08.999109    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:53:08.999209 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:08.999172    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs podName:4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:12.999155275 +0000 UTC m=+10.030021583 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs") pod "network-metrics-daemon-cbgw4" (UID: "4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:53:09.099912 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:09.099870    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret\") pod \"global-pull-secret-syncer-vjgrh\" (UID: \"b22d72c4-fb0e-4f78-9e7e-0550e6928616\") " pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:09.100090 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:09.100050    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 12:53:09.100171 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:09.100113    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret podName:b22d72c4-fb0e-4f78-9e7e-0550e6928616 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:13.100095662 +0000 UTC m=+10.130961960 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret") pod "global-pull-secret-syncer-vjgrh" (UID: "b22d72c4-fb0e-4f78-9e7e-0550e6928616") : object "kube-system"/"original-pull-secret" not registered
Sep 07 12:53:09.200645 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:09.200602    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvmx\" (UniqueName: \"kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx\") pod \"network-check-target-5tbdn\" (UID: \"fbbea119-b216-4988-ba0d-7c408dac7fc6\") " pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:09.200932 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:09.200910    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 12:53:09.201014 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:09.200941    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 12:53:09.201014 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:09.200954    2565 projected.go:196] Error preparing data for projected volume kube-api-access-5zvmx for pod openshift-network-diagnostics/network-check-target-5tbdn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:53:09.201114 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:09.201022    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx podName:fbbea119-b216-4988-ba0d-7c408dac7fc6 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:13.201001158 +0000 UTC m=+10.231867464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zvmx" (UniqueName: "kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx") pod "network-check-target-5tbdn" (UID: "fbbea119-b216-4988-ba0d-7c408dac7fc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:53:09.483844 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:09.483459    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:09.483844 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:09.483621    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:10.481653 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:10.481612    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:10.481817 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:10.481753    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:10.482192 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:10.482167    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:10.482320 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:10.482264    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:11.487049 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:11.486693    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:11.487049 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:11.486791    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:12.481874 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:12.481838    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:12.482037 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:12.481843    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:12.482037 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:12.481986    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:12.482136 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:12.482077    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:13.032370 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:13.032309    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs\") pod \"network-metrics-daemon-cbgw4\" (UID: \"4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4\") " pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:13.032860 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:13.032475    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:53:13.032860 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:13.032535    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs podName:4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:21.032514523 +0000 UTC m=+18.063380814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs") pod "network-metrics-daemon-cbgw4" (UID: "4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:53:13.132781 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:13.132696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret\") pod \"global-pull-secret-syncer-vjgrh\" (UID: \"b22d72c4-fb0e-4f78-9e7e-0550e6928616\") " pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:13.132963 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:13.132859    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 12:53:13.132963 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:13.132938    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret podName:b22d72c4-fb0e-4f78-9e7e-0550e6928616 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:21.132915872 +0000 UTC m=+18.163782178 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret") pod "global-pull-secret-syncer-vjgrh" (UID: "b22d72c4-fb0e-4f78-9e7e-0550e6928616") : object "kube-system"/"original-pull-secret" not registered
Sep 07 12:53:13.233797 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:13.233755    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvmx\" (UniqueName: \"kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx\") pod \"network-check-target-5tbdn\" (UID: \"fbbea119-b216-4988-ba0d-7c408dac7fc6\") " pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:13.233979 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:13.233878    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 12:53:13.233979 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:13.233891    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 12:53:13.233979 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:13.233900    2565 projected.go:196] Error preparing data for projected volume kube-api-access-5zvmx for pod openshift-network-diagnostics/network-check-target-5tbdn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:53:13.233979 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:13.233950    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx podName:fbbea119-b216-4988-ba0d-7c408dac7fc6 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:21.233937439 +0000 UTC m=+18.264803740 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zvmx" (UniqueName: "kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx") pod "network-check-target-5tbdn" (UID: "fbbea119-b216-4988-ba0d-7c408dac7fc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:53:13.482732 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:13.482657    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:13.482880 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:13.482785    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:14.482542 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:14.482495    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:14.483177 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:14.483143    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:14.483465 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:14.483426    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:14.483601 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:14.483575    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:15.481840 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:15.481806    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:15.482013 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:15.481962    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:16.482124 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:16.482088    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:16.482569 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:16.482197    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:16.482569 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:16.482254    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:16.482569 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:16.482369    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:17.482388 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:17.482350    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:17.482830 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:17.482514    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:18.482465 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:18.482430    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:18.482885 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:18.482557    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:18.482885 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:18.482632    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:18.482885 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:18.482754    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:19.481697 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:19.481652    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:19.481881 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:19.481808    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:20.482098 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:20.482016    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:20.482591 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:20.482018    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:20.482591 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:20.482141    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:20.482591 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:20.482240    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:21.091511 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:21.091480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs\") pod \"network-metrics-daemon-cbgw4\" (UID: \"4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4\") " pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:21.091675 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:21.091598    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:53:21.091675 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:21.091654    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs podName:4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:37.091637729 +0000 UTC m=+34.122504022 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs") pod "network-metrics-daemon-cbgw4" (UID: "4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:53:21.192253 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:21.192219    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret\") pod \"global-pull-secret-syncer-vjgrh\" (UID: \"b22d72c4-fb0e-4f78-9e7e-0550e6928616\") " pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:21.192432 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:21.192347    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 12:53:21.192432 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:21.192416    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret podName:b22d72c4-fb0e-4f78-9e7e-0550e6928616 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:37.19239787 +0000 UTC m=+34.223264174 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret") pod "global-pull-secret-syncer-vjgrh" (UID: "b22d72c4-fb0e-4f78-9e7e-0550e6928616") : object "kube-system"/"original-pull-secret" not registered
Sep 07 12:53:21.293548 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:21.293506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvmx\" (UniqueName: \"kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx\") pod \"network-check-target-5tbdn\" (UID: \"fbbea119-b216-4988-ba0d-7c408dac7fc6\") " pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:21.293719 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:21.293662    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 12:53:21.293719 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:21.293687    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 12:53:21.293719 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:21.293699    2565 projected.go:196] Error preparing data for projected volume kube-api-access-5zvmx for pod openshift-network-diagnostics/network-check-target-5tbdn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:53:21.293839 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:21.293755    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx podName:fbbea119-b216-4988-ba0d-7c408dac7fc6 nodeName:}" failed. No retries permitted until 2026-09-07 12:53:37.293741846 +0000 UTC m=+34.324608147 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zvmx" (UniqueName: "kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx") pod "network-check-target-5tbdn" (UID: "fbbea119-b216-4988-ba0d-7c408dac7fc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:53:21.482292 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:21.482191    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:21.482770 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:21.482378    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:22.481907 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:22.481872    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:22.482090 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:22.481986    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:22.482090 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:22.482054    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:22.482192 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:22.482166    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:23.488624 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:23.488423    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:23.488934 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:23.488749    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:23.560399 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:23.560365    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f5wl5" event={"ID":"a4c45d9f-f285-4600-b036-836ef644d234","Type":"ContainerStarted","Data":"7a9d4b9f4bfec5f9d754cd6066454acb1b8e4fe5739686168325b4d0aac42aa0"}
Sep 07 12:53:23.563955 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:23.563924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sjshd" event={"ID":"20fb9fad-025c-42bb-92e5-86b868e484b7","Type":"ContainerStarted","Data":"73817c5a8666f7efccfcdd46824d86cd872dcb7e47e5673066ab4f2edaa8ff99"}
Sep 07 12:53:23.566609 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:23.566092    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dqbpb" event={"ID":"aaac4f3c-dfa1-4c7f-8e67-2e01a910c117","Type":"ContainerStarted","Data":"a46365f5b9bf7055559b25fd5b12468ddd86f9ac55f3d03745f236bfb4e27519"}
Sep 07 12:53:23.570038 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:23.569808    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7gz9f" event={"ID":"d4450013-6c87-4e72-8722-20add36bb64a","Type":"ContainerStarted","Data":"eeebd312c4070395f59eb69243ebe641d3060e9021e342aacf7451f429849090"}
Sep 07 12:53:23.573795 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:23.573754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx" event={"ID":"6c743a37-043a-4a2a-8d5b-dc151cc29e58","Type":"ContainerStarted","Data":"8d4f04d1bd1b3d032cd27cfe5d471c0e0783f258e5b1df141e093885609f3771"}
Sep 07 12:53:23.575487 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:23.575240    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c458f" event={"ID":"d0a3de58-4067-45e4-b1b4-7cfce37b0fe4","Type":"ContainerStarted","Data":"9cfe215a30ac7dcf019ec5f9e371c606c82569bf356f7d6d6fdf5addabc9f459"}
Sep 07 12:53:23.577697 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:23.577345    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w7hq9" event={"ID":"da9073e8-463b-4ec5-b1e2-03a9260d2471","Type":"ContainerStarted","Data":"448406a423d430c8e10fe7e3a50356595624b904b1f53599e3153e84879a2aaf"}
Sep 07 12:53:23.582983 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:23.582954    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr" event={"ID":"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd","Type":"ContainerStarted","Data":"b2137f328a04742dafc6685cf9d4e0d862d492c3d050ca13f6ab6465c44f1632"}
Sep 07 12:53:23.681963 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:23.681892    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c458f" podStartSLOduration=3.867935573 podStartE2EDuration="20.681877445s" totalImagesPullingTime="16.813941872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:53:23.681675922 +0000 UTC m=+20.712542234" watchObservedRunningTime="2026-09-07 12:53:23.681877445 +0000 UTC m=+20.712743755"
Sep 07 12:53:23.682257 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:23.682234    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sjshd" podStartSLOduration=3.814299374 podStartE2EDuration="20.682225455s" totalImagesPullingTime="16.867926081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:53:23.643316926 +0000 UTC m=+20.674183247" watchObservedRunningTime="2026-09-07 12:53:23.682225455 +0000 UTC m=+20.713091765"
Sep 07 12:53:23.707959 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:23.707916    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dqbpb" podStartSLOduration=3.901379284 podStartE2EDuration="20.707901223s" totalImagesPullingTime="16.806521939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:53:23.707090453 +0000 UTC m=+20.737956763" watchObservedRunningTime="2026-09-07 12:53:23.707901223 +0000 UTC m=+20.738767533"
Sep 07 12:53:23.739990 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:23.739947    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-w7hq9" podStartSLOduration=8.248680049 podStartE2EDuration="20.739931643s" totalImagesPullingTime="12.491251594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:53:23.739901196 +0000 UTC m=+20.770767507" watchObservedRunningTime="2026-09-07 12:53:23.739931643 +0000 UTC m=+20.770797952"
Sep 07 12:53:23.764742 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:23.764696    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7gz9f" podStartSLOduration=3.950015265 podStartE2EDuration="20.764676668s" totalImagesPullingTime="16.814661403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:53:23.764555228 +0000 UTC m=+20.795421548" watchObservedRunningTime="2026-09-07 12:53:23.764676668 +0000 UTC m=+20.795543045"
Sep 07 12:53:24.481891 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:24.481862    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:24.482042 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:24.481874    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:24.482042 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:24.481988    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:24.482042 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:24.482024    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:24.589798 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:24.589712    2565 generic.go:356] "Generic (PLEG): container finished" podID="a4c45d9f-f285-4600-b036-836ef644d234" containerID="7a9d4b9f4bfec5f9d754cd6066454acb1b8e4fe5739686168325b4d0aac42aa0" exitCode=0
Sep 07 12:53:24.590613 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:24.589807    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f5wl5" event={"ID":"a4c45d9f-f285-4600-b036-836ef644d234","Type":"ContainerDied","Data":"7a9d4b9f4bfec5f9d754cd6066454acb1b8e4fe5739686168325b4d0aac42aa0"}
Sep 07 12:53:24.594173 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:24.594143    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx" event={"ID":"6c743a37-043a-4a2a-8d5b-dc151cc29e58","Type":"ContainerStarted","Data":"0b0071169550ccab87975318082e626efe5e1a8a625aec77172bf3134d2d99f3"}
Sep 07 12:53:24.594266 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:24.594233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx" event={"ID":"6c743a37-043a-4a2a-8d5b-dc151cc29e58","Type":"ContainerStarted","Data":"bc18636513bcd714e428210c5dea7bd329b098ea4a4fd6cd3b37862c85c008cf"}
Sep 07 12:53:24.594322 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:24.594297    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx" event={"ID":"6c743a37-043a-4a2a-8d5b-dc151cc29e58","Type":"ContainerStarted","Data":"aceb83dcc0bd52b05416e06d35ae1eea78c2b21e52c184eab65401f82f04311b"}
Sep 07 12:53:24.594386 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:24.594370    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx" event={"ID":"6c743a37-043a-4a2a-8d5b-dc151cc29e58","Type":"ContainerStarted","Data":"77fef6d6132e52ef2be36d9ce6525a9293e4befa4c4ce13e9fbf61f7bcc61ed3"}
Sep 07 12:53:24.594429 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:24.594421    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx" event={"ID":"6c743a37-043a-4a2a-8d5b-dc151cc29e58","Type":"ContainerStarted","Data":"bc4e756c1245ca55c75f4ab7917e3db5be5b8c81f3370213f0101ea4dae17bbe"}
Sep 07 12:53:24.766506 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:24.766481    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 07 12:53:25.422409 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:25.422267    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T12:53:24.766503065Z","UUID":"4153c291-d7ea-4194-97bc-112ce31909d1","Handler":null,"Name":"","Endpoint":""}
Sep 07 12:53:25.424127 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:25.424100    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 07 12:53:25.424256 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:25.424146    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 07 12:53:25.482475 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:25.482444    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:25.482629 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:25.482565    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:25.598286 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:25.598251    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr" event={"ID":"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd","Type":"ContainerStarted","Data":"6fa98024cfd9a21635dfc1ff48823dce50d4e24865a07448bf02278495ebd4d3"}
Sep 07 12:53:25.599829 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:25.599803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9jfkw" event={"ID":"93cdd73f-1603-4e5a-bde9-e5e967b168cd","Type":"ContainerStarted","Data":"20abab16e18eef15b85c3f4b2f6287aa837373babc29ab4b1bc1ec81237779a9"}
Sep 07 12:53:25.617832 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:25.617497    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9jfkw" podStartSLOduration=5.774052201 podStartE2EDuration="22.617478087s" totalImagesPullingTime="16.843425886s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:53:25.616787893 +0000 UTC m=+22.647654214" watchObservedRunningTime="2026-09-07 12:53:25.617478087 +0000 UTC m=+22.648344398"
Sep 07 12:53:26.482539 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:26.482504    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:26.482539 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:26.482541    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:26.482869 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:26.482614    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:26.482869 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:26.482691    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:26.605248 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:26.605206    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr" event={"ID":"efb6fcfc-8f7a-4157-a56e-6cbd63d931bd","Type":"ContainerStarted","Data":"b95947440ee18b3269f7b5acc056d3890d0c1f7f6526667da763ebad64ea185d"}
Sep 07 12:53:26.609113 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:26.609082    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx" event={"ID":"6c743a37-043a-4a2a-8d5b-dc151cc29e58","Type":"ContainerStarted","Data":"305fcd9c556824b7a7dc541bd46fd7c03e4436ef9ba2317a8fc62b8d1cfe1933"}
Sep 07 12:53:26.627419 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:26.627379    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9z9gr" podStartSLOduration=4.856898542 podStartE2EDuration="23.627363563s" totalImagesPullingTime="18.770465021s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:53:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:53:26.626856107 +0000 UTC m=+23.657722429" watchObservedRunningTime="2026-09-07 12:53:26.627363563 +0000 UTC m=+23.658229872"
Sep 07 12:53:26.757396 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:26.757301    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-w7hq9"
Sep 07 12:53:26.757832 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:26.757813    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-w7hq9"
Sep 07 12:53:27.482802 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:27.482578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:27.482966 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:27.482917    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:27.611731 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:27.611684    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-w7hq9"
Sep 07 12:53:27.612158 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:27.611973    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w7hq9"
Sep 07 12:53:28.482232 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:28.482205    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:28.482420 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:28.482205    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:28.482420 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:28.482318    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:28.482525 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:28.482430    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:29.482244 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:29.482034    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:29.483064 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:29.482389    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:29.617008 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:29.616976    2565 generic.go:356] "Generic (PLEG): container finished" podID="a4c45d9f-f285-4600-b036-836ef644d234" containerID="96fae54d2e2f5d2b94e903819af57355a7deb0eaef54e8774db2f61b0991000a" exitCode=0
Sep 07 12:53:29.617173 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:29.617059    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f5wl5" event={"ID":"a4c45d9f-f285-4600-b036-836ef644d234","Type":"ContainerDied","Data":"96fae54d2e2f5d2b94e903819af57355a7deb0eaef54e8774db2f61b0991000a"}
Sep 07 12:53:29.620963 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:29.620939    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx" event={"ID":"6c743a37-043a-4a2a-8d5b-dc151cc29e58","Type":"ContainerStarted","Data":"9337568aa196ae8b66feab8b8c3260cb39b1f792adc04f0c0fe92ede5ad32896"}
Sep 07 12:53:29.621195 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:29.621180    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:29.621276 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:29.621247    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:29.637208 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:29.637131    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:29.685810 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:29.685760    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx" podStartSLOduration=9.735199639 podStartE2EDuration="26.685745939s" totalImagesPullingTime="16.9505463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:53:29.684445461 +0000 UTC m=+26.715311773" watchObservedRunningTime="2026-09-07 12:53:29.685745939 +0000 UTC m=+26.716612250"
Sep 07 12:53:30.481899 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:30.481816    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:30.481899 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:30.481851    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:30.482163 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:30.481970    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:30.482509 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:30.482459    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:30.623966 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:30.623921    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:30.640702 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:30.640682    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:53:31.109708 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:31.109602    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vjgrh"]
Sep 07 12:53:31.109828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:31.109774    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:31.109921 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:31.109893    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:31.110168 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:31.110144    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cbgw4"]
Sep 07 12:53:31.110283 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:31.110274    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:31.110409 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:31.110390    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:31.110717 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:31.110697    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5tbdn"]
Sep 07 12:53:31.110793 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:31.110781    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:31.110895 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:31.110871    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:31.627585 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:31.627544    2565 generic.go:356] "Generic (PLEG): container finished" podID="a4c45d9f-f285-4600-b036-836ef644d234" containerID="8342d6b4851d241c4862066d31b6d9423163996ce99f06251a27c1f2a7b5dfba" exitCode=0
Sep 07 12:53:31.628047 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:31.627635    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f5wl5" event={"ID":"a4c45d9f-f285-4600-b036-836ef644d234","Type":"ContainerDied","Data":"8342d6b4851d241c4862066d31b6d9423163996ce99f06251a27c1f2a7b5dfba"}
Sep 07 12:53:32.392992 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:32.392963    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dqbpb_aaac4f3c-dfa1-4c7f-8e67-2e01a910c117/dns-node-resolver/0.log"
Sep 07 12:53:32.482202 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:32.482029    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:32.482365 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:32.482082    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:32.482365 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:32.482282    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:32.482432 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:32.482392    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:32.632106 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:32.632027    2565 generic.go:356] "Generic (PLEG): container finished" podID="a4c45d9f-f285-4600-b036-836ef644d234" containerID="51124c0d83d3ee284e2274e2e5fdd959fd504dce346d9ff5b127b31cc3fe484b" exitCode=0
Sep 07 12:53:32.632448 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:32.632105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f5wl5" event={"ID":"a4c45d9f-f285-4600-b036-836ef644d234","Type":"ContainerDied","Data":"51124c0d83d3ee284e2274e2e5fdd959fd504dce346d9ff5b127b31cc3fe484b"}
Sep 07 12:53:33.143471 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:33.143440    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7gz9f_d4450013-6c87-4e72-8722-20add36bb64a/node-ca/0.log"
Sep 07 12:53:33.482919 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:33.482843    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:33.483076 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:33.482932    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:34.481776 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:34.481701    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:34.482131 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:34.481799    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:34.482131 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:34.481853    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:34.482131 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:34.481930    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:35.482576 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:35.482543    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:35.483029 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:35.482685    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:36.481652 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:36.481576    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:36.481800 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:36.481579    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:36.481800 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:36.481683    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:36.481800 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:36.481755    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:37.107673 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:37.107633    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs\") pod \"network-metrics-daemon-cbgw4\" (UID: \"4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4\") " pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:37.108259 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:37.107786    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:53:37.108259 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:37.107856    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs podName:4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4 nodeName:}" failed. No retries permitted until 2026-09-07 12:54:09.107836058 +0000 UTC m=+66.138702396 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs") pod "network-metrics-daemon-cbgw4" (UID: "4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 12:53:37.208063 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:37.208023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret\") pod \"global-pull-secret-syncer-vjgrh\" (UID: \"b22d72c4-fb0e-4f78-9e7e-0550e6928616\") " pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:37.208239 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:37.208217    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 12:53:37.208364 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:37.208344    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret podName:b22d72c4-fb0e-4f78-9e7e-0550e6928616 nodeName:}" failed. No retries permitted until 2026-09-07 12:54:09.208314577 +0000 UTC m=+66.239180877 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret") pod "global-pull-secret-syncer-vjgrh" (UID: "b22d72c4-fb0e-4f78-9e7e-0550e6928616") : object "kube-system"/"original-pull-secret" not registered
Sep 07 12:53:37.309492 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:37.309448    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvmx\" (UniqueName: \"kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx\") pod \"network-check-target-5tbdn\" (UID: \"fbbea119-b216-4988-ba0d-7c408dac7fc6\") " pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:37.309659 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:37.309641    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 12:53:37.309710 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:37.309670    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 12:53:37.309710 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:37.309684    2565 projected.go:196] Error preparing data for projected volume kube-api-access-5zvmx for pod openshift-network-diagnostics/network-check-target-5tbdn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:53:37.309786 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:37.309775    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx podName:fbbea119-b216-4988-ba0d-7c408dac7fc6 nodeName:}" failed. No retries permitted until 2026-09-07 12:54:09.309760554 +0000 UTC m=+66.340626845 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zvmx" (UniqueName: "kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx") pod "network-check-target-5tbdn" (UID: "fbbea119-b216-4988-ba0d-7c408dac7fc6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 12:53:37.481791 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:37.481711    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:37.481946 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:37.481850    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:38.482379 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:38.482341    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:38.482835 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:38.482389    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:38.482835 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:38.482452    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:38.482835 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:38.482556    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:39.482321 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:39.482286    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:39.482526 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:39.482451    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:39.650482 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:39.650453    2565 generic.go:356] "Generic (PLEG): container finished" podID="a4c45d9f-f285-4600-b036-836ef644d234" containerID="d8bd4558551a8971f0395a03331bfed1c217190d5063e2945e5fd633d73af521" exitCode=0
Sep 07 12:53:39.650638 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:39.650493    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f5wl5" event={"ID":"a4c45d9f-f285-4600-b036-836ef644d234","Type":"ContainerDied","Data":"d8bd4558551a8971f0395a03331bfed1c217190d5063e2945e5fd633d73af521"}
Sep 07 12:53:40.481989 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:40.481913    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:40.482138 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:40.481924    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:40.482138 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:40.482037    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:40.482138 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:40.482083    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:40.655584 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:40.655554    2565 generic.go:356] "Generic (PLEG): container finished" podID="a4c45d9f-f285-4600-b036-836ef644d234" containerID="d9c4d393aa2fd2c8737bd8e45cff0ecb208df0285259abd1638c5baa61507931" exitCode=0
Sep 07 12:53:40.655993 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:40.655609    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f5wl5" event={"ID":"a4c45d9f-f285-4600-b036-836ef644d234","Type":"ContainerDied","Data":"d9c4d393aa2fd2c8737bd8e45cff0ecb208df0285259abd1638c5baa61507931"}
Sep 07 12:53:41.481603 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:41.481576    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:41.481779 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:41.481705    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:41.660881 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:41.660844    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f5wl5" event={"ID":"a4c45d9f-f285-4600-b036-836ef644d234","Type":"ContainerStarted","Data":"658f1bd267ca8e5d9e1ec79a0e1e3690bd4bd165de23eb715058db0c86250ec7"}
Sep 07 12:53:41.686441 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:41.686387    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-f5wl5" podStartSLOduration=9.452593791 podStartE2EDuration="38.686370727s" totalImagesPullingTime="29.233776936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:53:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:53:41.685194213 +0000 UTC m=+38.716060523" watchObservedRunningTime="2026-09-07 12:53:41.686370727 +0000 UTC m=+38.717237036"
Sep 07 12:53:42.481546 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:42.481501    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:42.481749 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:42.481504    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:42.481749 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:42.481610    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:42.481749 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:42.481708    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:43.483312 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:43.483280    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:43.483739 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:43.483410    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:44.481955 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:44.481920    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:44.482132 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:44.481920    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:44.482132 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:44.482022    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:44.482132 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:44.482087    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:45.481738 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:45.481701    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:45.482080 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:45.481813    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:46.482043 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:46.482014    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:46.482446 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:46.482014    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:46.482446 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:46.482116    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:46.482446 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:46.482177    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:47.482132 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:47.482097    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:47.482518 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:47.482201    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:48.482203 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:48.482123    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:48.482203 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:48.482152    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:48.482640 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:48.482242    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:48.482640 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:48.482380    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:49.482523 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:49.482490    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:49.482900 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:49.482598    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:50.482154 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:50.482122    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:50.482350 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:50.482124    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:50.482350 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:50.482239    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:50.482350 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:50.482309    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:51.481747 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:51.481712    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:51.482168 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:51.481850    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:52.481730 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:52.481696    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:52.481908 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:52.481808    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:52.481908 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:52.481860    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:52.482190 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:52.481942    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:53.482905 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:53.482872    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:53.483429 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:53.482997    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cbgw4" podUID="4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4"
Sep 07 12:53:54.481632 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:54.481595    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:54.481828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:54.481599    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:54.481828 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:54.481724    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjgrh" podUID="b22d72c4-fb0e-4f78-9e7e-0550e6928616"
Sep 07 12:53:54.481828 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:53:54.481772    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tbdn" podUID="fbbea119-b216-4988-ba0d-7c408dac7fc6"
Sep 07 12:53:55.091475 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.091385    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeReady"
Sep 07 12:53:55.091910 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.091532    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 07 12:53:55.149535 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.149498    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sb8hw"]
Sep 07 12:53:55.151075 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.151054    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-822q8"]
Sep 07 12:53:55.151202 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.151183    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sb8hw"
Sep 07 12:53:55.152750 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.152728    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.154122 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.154101    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 07 12:53:55.154254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.154230    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8c6z8\""
Sep 07 12:53:55.154719 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.154705    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 07 12:53:55.155137 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.155120    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 07 12:53:55.155253 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.155240    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 07 12:53:55.155430 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.155396    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 07 12:53:55.155535 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.155469    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6fwj5\""
Sep 07 12:53:55.155685 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.155669    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 07 12:53:55.165617 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.165588    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sb8hw"]
Sep 07 12:53:55.168555 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.168532    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-822q8"]
Sep 07 12:53:55.237847 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.237811    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/22574e81-953a-49b3-8305-6456988043d0-config-volume\") pod \"dns-default-sb8hw\" (UID: \"22574e81-953a-49b3-8305-6456988043d0\") " pod="openshift-dns/dns-default-sb8hw"
Sep 07 12:53:55.237847 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.237853    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22574e81-953a-49b3-8305-6456988043d0-tmp-dir\") pod \"dns-default-sb8hw\" (UID: \"22574e81-953a-49b3-8305-6456988043d0\") " pod="openshift-dns/dns-default-sb8hw"
Sep 07 12:53:55.238095 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.237886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08e62ed2-77a2-4656-afad-a44e252f2ede-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-822q8\" (UID: \"08e62ed2-77a2-4656-afad-a44e252f2ede\") " pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.238095 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.237910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvtjc\" (UniqueName: \"kubernetes.io/projected/22574e81-953a-49b3-8305-6456988043d0-kube-api-access-nvtjc\") pod \"dns-default-sb8hw\" (UID: \"22574e81-953a-49b3-8305-6456988043d0\") " pod="openshift-dns/dns-default-sb8hw"
Sep 07 12:53:55.238095 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.237964    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08e62ed2-77a2-4656-afad-a44e252f2ede-crio-socket\") pod \"insights-runtime-extractor-822q8\" (UID: \"08e62ed2-77a2-4656-afad-a44e252f2ede\") " pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.238095 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.238002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08e62ed2-77a2-4656-afad-a44e252f2ede-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-822q8\" (UID: \"08e62ed2-77a2-4656-afad-a44e252f2ede\") " pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.238095 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.238030    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lg5b\" (UniqueName: \"kubernetes.io/projected/08e62ed2-77a2-4656-afad-a44e252f2ede-kube-api-access-2lg5b\") pod \"insights-runtime-extractor-822q8\" (UID: \"08e62ed2-77a2-4656-afad-a44e252f2ede\") " pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.238095 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.238054    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22574e81-953a-49b3-8305-6456988043d0-metrics-tls\") pod \"dns-default-sb8hw\" (UID: \"22574e81-953a-49b3-8305-6456988043d0\") " pod="openshift-dns/dns-default-sb8hw"
Sep 07 12:53:55.238095 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.238079    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08e62ed2-77a2-4656-afad-a44e252f2ede-data-volume\") pod \"insights-runtime-extractor-822q8\" (UID: \"08e62ed2-77a2-4656-afad-a44e252f2ede\") " pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.244658 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.244635    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-brzmx"]
Sep 07 12:53:55.246178 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.246165    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-brzmx"
Sep 07 12:53:55.249087 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.249066    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 07 12:53:55.249448 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.249432    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j64n2\""
Sep 07 12:53:55.249517 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.249500    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 07 12:53:55.249626 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.249611    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 07 12:53:55.257307 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.257283    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-brzmx"]
Sep 07 12:53:55.338995 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.338966    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08e62ed2-77a2-4656-afad-a44e252f2ede-crio-socket\") pod \"insights-runtime-extractor-822q8\" (UID: \"08e62ed2-77a2-4656-afad-a44e252f2ede\") " pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.338995 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.338998    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08e62ed2-77a2-4656-afad-a44e252f2ede-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-822q8\" (UID: \"08e62ed2-77a2-4656-afad-a44e252f2ede\") " pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.339247 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.339018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lg5b\" (UniqueName: \"kubernetes.io/projected/08e62ed2-77a2-4656-afad-a44e252f2ede-kube-api-access-2lg5b\") pod \"insights-runtime-extractor-822q8\" (UID: \"08e62ed2-77a2-4656-afad-a44e252f2ede\") " pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.339247 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.339035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22574e81-953a-49b3-8305-6456988043d0-metrics-tls\") pod \"dns-default-sb8hw\" (UID: \"22574e81-953a-49b3-8305-6456988043d0\") " pod="openshift-dns/dns-default-sb8hw"
Sep 07 12:53:55.339247 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.339050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08e62ed2-77a2-4656-afad-a44e252f2ede-data-volume\") pod \"insights-runtime-extractor-822q8\" (UID: \"08e62ed2-77a2-4656-afad-a44e252f2ede\") " pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.339247 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.339082    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jh6f\" (UniqueName: \"kubernetes.io/projected/9a25b578-db78-49e9-b2d2-fb9440eef833-kube-api-access-5jh6f\") pod \"ingress-canary-brzmx\" (UID: \"9a25b578-db78-49e9-b2d2-fb9440eef833\") " pod="openshift-ingress-canary/ingress-canary-brzmx"
Sep 07 12:53:55.339247 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.339119    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/22574e81-953a-49b3-8305-6456988043d0-config-volume\") pod \"dns-default-sb8hw\" (UID: \"22574e81-953a-49b3-8305-6456988043d0\") " pod="openshift-dns/dns-default-sb8hw"
Sep 07 12:53:55.339247 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.339144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22574e81-953a-49b3-8305-6456988043d0-tmp-dir\") pod \"dns-default-sb8hw\" (UID: \"22574e81-953a-49b3-8305-6456988043d0\") " pod="openshift-dns/dns-default-sb8hw"
Sep 07 12:53:55.339247 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.339171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08e62ed2-77a2-4656-afad-a44e252f2ede-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-822q8\" (UID: \"08e62ed2-77a2-4656-afad-a44e252f2ede\") " pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.339247 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.339194    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a25b578-db78-49e9-b2d2-fb9440eef833-cert\") pod \"ingress-canary-brzmx\" (UID: \"9a25b578-db78-49e9-b2d2-fb9440eef833\") " pod="openshift-ingress-canary/ingress-canary-brzmx"
Sep 07 12:53:55.339247 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.339214    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08e62ed2-77a2-4656-afad-a44e252f2ede-crio-socket\") pod \"insights-runtime-extractor-822q8\" (UID: \"08e62ed2-77a2-4656-afad-a44e252f2ede\") " pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.339247 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.339232    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvtjc\" (UniqueName: \"kubernetes.io/projected/22574e81-953a-49b3-8305-6456988043d0-kube-api-access-nvtjc\") pod \"dns-default-sb8hw\" (UID: \"22574e81-953a-49b3-8305-6456988043d0\") " pod="openshift-dns/dns-default-sb8hw"
Sep 07 12:53:55.339773 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.339515    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08e62ed2-77a2-4656-afad-a44e252f2ede-data-volume\") pod \"insights-runtime-extractor-822q8\" (UID: \"08e62ed2-77a2-4656-afad-a44e252f2ede\") " pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.339773 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.339543    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22574e81-953a-49b3-8305-6456988043d0-tmp-dir\") pod \"dns-default-sb8hw\" (UID: \"22574e81-953a-49b3-8305-6456988043d0\") " pod="openshift-dns/dns-default-sb8hw"
Sep 07 12:53:55.339878 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.339814    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/22574e81-953a-49b3-8305-6456988043d0-config-volume\") pod \"dns-default-sb8hw\" (UID: \"22574e81-953a-49b3-8305-6456988043d0\") " pod="openshift-dns/dns-default-sb8hw"
Sep 07 12:53:55.339967 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.339947    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08e62ed2-77a2-4656-afad-a44e252f2ede-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-822q8\" (UID: \"08e62ed2-77a2-4656-afad-a44e252f2ede\") " pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.343138 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.343088    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08e62ed2-77a2-4656-afad-a44e252f2ede-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-822q8\" (UID: \"08e62ed2-77a2-4656-afad-a44e252f2ede\") " pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.343138 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.343123    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22574e81-953a-49b3-8305-6456988043d0-metrics-tls\") pod \"dns-default-sb8hw\" (UID: \"22574e81-953a-49b3-8305-6456988043d0\") " pod="openshift-dns/dns-default-sb8hw"
Sep 07 12:53:55.348658 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.348630    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lg5b\" (UniqueName: \"kubernetes.io/projected/08e62ed2-77a2-4656-afad-a44e252f2ede-kube-api-access-2lg5b\") pod \"insights-runtime-extractor-822q8\" (UID: \"08e62ed2-77a2-4656-afad-a44e252f2ede\") " pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.349111 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.349094    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvtjc\" (UniqueName: \"kubernetes.io/projected/22574e81-953a-49b3-8305-6456988043d0-kube-api-access-nvtjc\") pod \"dns-default-sb8hw\" (UID: \"22574e81-953a-49b3-8305-6456988043d0\") " pod="openshift-dns/dns-default-sb8hw"
Sep 07 12:53:55.440061 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.440026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jh6f\" (UniqueName: \"kubernetes.io/projected/9a25b578-db78-49e9-b2d2-fb9440eef833-kube-api-access-5jh6f\") pod \"ingress-canary-brzmx\" (UID: \"9a25b578-db78-49e9-b2d2-fb9440eef833\") " pod="openshift-ingress-canary/ingress-canary-brzmx"
Sep 07 12:53:55.440207 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.440075    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a25b578-db78-49e9-b2d2-fb9440eef833-cert\") pod \"ingress-canary-brzmx\" (UID: \"9a25b578-db78-49e9-b2d2-fb9440eef833\") " pod="openshift-ingress-canary/ingress-canary-brzmx"
Sep 07 12:53:55.442506 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.442488    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a25b578-db78-49e9-b2d2-fb9440eef833-cert\") pod \"ingress-canary-brzmx\" (UID: \"9a25b578-db78-49e9-b2d2-fb9440eef833\") " pod="openshift-ingress-canary/ingress-canary-brzmx"
Sep 07 12:53:55.449293 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.449264    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jh6f\" (UniqueName: \"kubernetes.io/projected/9a25b578-db78-49e9-b2d2-fb9440eef833-kube-api-access-5jh6f\") pod \"ingress-canary-brzmx\" (UID: \"9a25b578-db78-49e9-b2d2-fb9440eef833\") " pod="openshift-ingress-canary/ingress-canary-brzmx"
Sep 07 12:53:55.462101 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.462080    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sb8hw"
Sep 07 12:53:55.467187 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.467171    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-822q8"
Sep 07 12:53:55.481804 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.481784    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:53:55.484659 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.484641    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 12:53:55.484948 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.484933    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gnfpv\""
Sep 07 12:53:55.554421 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.554393    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-brzmx"
Sep 07 12:53:55.746430 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.746400    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sb8hw"]
Sep 07 12:53:55.748632 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.748454    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-822q8"]
Sep 07 12:53:55.751314 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:55.751261    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22574e81_953a_49b3_8305_6456988043d0.slice/crio-79429b5869c1399f05389660b062d2e99f3de3df703918c97c188fbe2c7eb6e2 WatchSource:0}: Error finding container 79429b5869c1399f05389660b062d2e99f3de3df703918c97c188fbe2c7eb6e2: Status 404 returned error can't find the container with id 79429b5869c1399f05389660b062d2e99f3de3df703918c97c188fbe2c7eb6e2
Sep 07 12:53:55.754043 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:55.754018    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e62ed2_77a2_4656_afad_a44e252f2ede.slice/crio-735871a6ba46ab591f0db23455f21505112ad6a2cdef481f7fde7fb5120ed7fa WatchSource:0}: Error finding container 735871a6ba46ab591f0db23455f21505112ad6a2cdef481f7fde7fb5120ed7fa: Status 404 returned error can't find the container with id 735871a6ba46ab591f0db23455f21505112ad6a2cdef481f7fde7fb5120ed7fa
Sep 07 12:53:55.827397 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:55.827367    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-brzmx"]
Sep 07 12:53:55.830804 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:53:55.830774    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a25b578_db78_49e9_b2d2_fb9440eef833.slice/crio-a126551fd46de498a572274c1001728adf764a17f277d6d1adb544cad407b057 WatchSource:0}: Error finding container a126551fd46de498a572274c1001728adf764a17f277d6d1adb544cad407b057: Status 404 returned error can't find the container with id a126551fd46de498a572274c1001728adf764a17f277d6d1adb544cad407b057
Sep 07 12:53:56.481670 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:56.481625    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:53:56.482071 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:56.481635    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:53:56.488542 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:56.488518    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 12:53:56.491630 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:56.490867    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 12:53:56.491630 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:56.491096    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xfxxc\""
Sep 07 12:53:56.491630 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:56.491164    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 12:53:56.698952 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:56.698911    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sb8hw" event={"ID":"22574e81-953a-49b3-8305-6456988043d0","Type":"ContainerStarted","Data":"79429b5869c1399f05389660b062d2e99f3de3df703918c97c188fbe2c7eb6e2"}
Sep 07 12:53:56.700546 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:56.700510    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-brzmx" event={"ID":"9a25b578-db78-49e9-b2d2-fb9440eef833","Type":"ContainerStarted","Data":"a126551fd46de498a572274c1001728adf764a17f277d6d1adb544cad407b057"}
Sep 07 12:53:56.702648 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:56.702614    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-822q8" event={"ID":"08e62ed2-77a2-4656-afad-a44e252f2ede","Type":"ContainerStarted","Data":"ac14b4176cdf2e249d957a4cb1d699da73a4822e7efbf51cf5da94d106e952de"}
Sep 07 12:53:56.702767 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:56.702679    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-822q8" event={"ID":"08e62ed2-77a2-4656-afad-a44e252f2ede","Type":"ContainerStarted","Data":"36349444c7de75b8c37087bcf1ea9778a5b6fb6a69eb6a6207dc68b78b4db230"}
Sep 07 12:53:56.702767 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:56.702709    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-822q8" event={"ID":"08e62ed2-77a2-4656-afad-a44e252f2ede","Type":"ContainerStarted","Data":"735871a6ba46ab591f0db23455f21505112ad6a2cdef481f7fde7fb5120ed7fa"}
Sep 07 12:53:59.572946 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.572911    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-js568"]
Sep 07 12:53:59.575618 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.575603    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-js568"
Sep 07 12:53:59.578847 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.578823    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 07 12:53:59.578847 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.578841    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xkx6v\""
Sep 07 12:53:59.588508 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.588479    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-js568"]
Sep 07 12:53:59.669987 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.669950    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/13de3759-1c38-4740-8eff-496554cb5525-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-js568\" (UID: \"13de3759-1c38-4740-8eff-496554cb5525\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-js568"
Sep 07 12:53:59.715590 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.715556    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sb8hw" event={"ID":"22574e81-953a-49b3-8305-6456988043d0","Type":"ContainerStarted","Data":"892fbcdac1119ebd31526139b03d8dae3f59aa56154f6ff5b1045d5efda6b3f7"}
Sep 07 12:53:59.715768 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.715609    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sb8hw" event={"ID":"22574e81-953a-49b3-8305-6456988043d0","Type":"ContainerStarted","Data":"090489fb462cc4248d785dc9721d3eb56f84fdf02c7e485d688e651b39b8bf09"}
Sep 07 12:53:59.715768 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.715638    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sb8hw"
Sep 07 12:53:59.717126 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.717105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-brzmx" event={"ID":"9a25b578-db78-49e9-b2d2-fb9440eef833","Type":"ContainerStarted","Data":"0e5f8e9eedfe2ec4b9509e35601ecb3a4185b408129a4bdb6bcd80e5909bd550"}
Sep 07 12:53:59.719217 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.719194    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-822q8" event={"ID":"08e62ed2-77a2-4656-afad-a44e252f2ede","Type":"ContainerStarted","Data":"ea5df8fbb4bd2e6285967b78636cdcd51bc4aeeac813d4f0aab4f2ce1347f145"}
Sep 07 12:53:59.741431 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.741386    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sb8hw" podStartSLOduration=1.719375387 podStartE2EDuration="4.741371175s" totalImagesPullingTime="3.021995788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:53:59.7404481 +0000 UTC m=+56.771314410" watchObservedRunningTime="2026-09-07 12:53:59.741371175 +0000 UTC m=+56.772237484"
Sep 07 12:53:59.760797 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.760752    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-822q8" podStartSLOduration=1.870903997 podStartE2EDuration="4.760737079s" totalImagesPullingTime="2.889833082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:53:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:53:59.760157094 +0000 UTC m=+56.791023403" watchObservedRunningTime="2026-09-07 12:53:59.760737079 +0000 UTC m=+56.791603388"
Sep 07 12:53:59.770895 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.770863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/13de3759-1c38-4740-8eff-496554cb5525-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-js568\" (UID: \"13de3759-1c38-4740-8eff-496554cb5525\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-js568"
Sep 07 12:53:59.773464 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.773442    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/13de3759-1c38-4740-8eff-496554cb5525-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-js568\" (UID: \"13de3759-1c38-4740-8eff-496554cb5525\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-js568"
Sep 07 12:53:59.778067 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.778025    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-brzmx" podStartSLOduration=1.830595823 podStartE2EDuration="4.778010186s" totalImagesPullingTime="2.947414363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:53:59.777603346 +0000 UTC m=+56.808469656" watchObservedRunningTime="2026-09-07 12:53:59.778010186 +0000 UTC m=+56.808876530"
Sep 07 12:53:59.884610 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:53:59.884536    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-js568"
Sep 07 12:54:00.151950 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:00.151842    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-js568"]
Sep 07 12:54:00.156651 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:54:00.156622    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13de3759_1c38_4740_8eff_496554cb5525.slice/crio-098876c49e23a62e656719b266966cac63845439a40821b3c52b1e3037bc8184 WatchSource:0}: Error finding container 098876c49e23a62e656719b266966cac63845439a40821b3c52b1e3037bc8184: Status 404 returned error can't find the container with id 098876c49e23a62e656719b266966cac63845439a40821b3c52b1e3037bc8184
Sep 07 12:54:00.723858 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:00.723823    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-js568" event={"ID":"13de3759-1c38-4740-8eff-496554cb5525","Type":"ContainerStarted","Data":"098876c49e23a62e656719b266966cac63845439a40821b3c52b1e3037bc8184"}
Sep 07 12:54:02.643683 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:02.643653    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-smjnx"
Sep 07 12:54:02.731579 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:02.731549    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-js568" event={"ID":"13de3759-1c38-4740-8eff-496554cb5525","Type":"ContainerStarted","Data":"2de08ae70e1cb849563f51a68b21126c5f73f4f366cd3e31aace17a8a2d7ecf5"}
Sep 07 12:54:02.731771 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:02.731740    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-js568"
Sep 07 12:54:02.736446 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:02.736415    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-js568"
Sep 07 12:54:02.748117 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:02.748061    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-js568" podStartSLOduration=2.115195751 podStartE2EDuration="3.748045s" totalImagesPullingTime="1.632849249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:54:02.747608989 +0000 UTC m=+59.778475311" watchObservedRunningTime="2026-09-07 12:54:02.748045 +0000 UTC m=+59.778911310"
Sep 07 12:54:03.655037 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.654997    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-z45nq"]
Sep 07 12:54:03.656939 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.656924    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq"
Sep 07 12:54:03.661880 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.661855    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 07 12:54:03.662018 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.661935    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 07 12:54:03.662979 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.662956    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 07 12:54:03.662979 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.662972    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ptnwl\""
Sep 07 12:54:03.663146 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.662960    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 07 12:54:03.663146 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.663064    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 07 12:54:03.673910 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.673885    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-z45nq"]
Sep 07 12:54:03.697951 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.697919    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8df4290a-39f3-4ea5-b30e-6ccd32e57bca-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-z45nq\" (UID: \"8df4290a-39f3-4ea5-b30e-6ccd32e57bca\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq"
Sep 07 12:54:03.697951 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.697951    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8df4290a-39f3-4ea5-b30e-6ccd32e57bca-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-z45nq\" (UID: \"8df4290a-39f3-4ea5-b30e-6ccd32e57bca\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq"
Sep 07 12:54:03.698143 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.697970    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48xlb\" (UniqueName: \"kubernetes.io/projected/8df4290a-39f3-4ea5-b30e-6ccd32e57bca-kube-api-access-48xlb\") pod \"prometheus-operator-6f4946db56-z45nq\" (UID: \"8df4290a-39f3-4ea5-b30e-6ccd32e57bca\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq"
Sep 07 12:54:03.698143 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.698043    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8df4290a-39f3-4ea5-b30e-6ccd32e57bca-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-z45nq\" (UID: \"8df4290a-39f3-4ea5-b30e-6ccd32e57bca\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq"
Sep 07 12:54:03.798638 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.798580    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8df4290a-39f3-4ea5-b30e-6ccd32e57bca-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-z45nq\" (UID: \"8df4290a-39f3-4ea5-b30e-6ccd32e57bca\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq"
Sep 07 12:54:03.798638 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.798633    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48xlb\" (UniqueName: \"kubernetes.io/projected/8df4290a-39f3-4ea5-b30e-6ccd32e57bca-kube-api-access-48xlb\") pod \"prometheus-operator-6f4946db56-z45nq\" (UID: \"8df4290a-39f3-4ea5-b30e-6ccd32e57bca\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq"
Sep 07 12:54:03.798831 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.798677    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8df4290a-39f3-4ea5-b30e-6ccd32e57bca-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-z45nq\" (UID: \"8df4290a-39f3-4ea5-b30e-6ccd32e57bca\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq"
Sep 07 12:54:03.798831 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.798727    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8df4290a-39f3-4ea5-b30e-6ccd32e57bca-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-z45nq\" (UID: \"8df4290a-39f3-4ea5-b30e-6ccd32e57bca\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq"
Sep 07 12:54:03.798917 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:54:03.798828    2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 07 12:54:03.798917 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:54:03.798911    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8df4290a-39f3-4ea5-b30e-6ccd32e57bca-prometheus-operator-tls podName:8df4290a-39f3-4ea5-b30e-6ccd32e57bca nodeName:}" failed. No retries permitted until 2026-09-07 12:54:04.298891949 +0000 UTC m=+61.329758239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/8df4290a-39f3-4ea5-b30e-6ccd32e57bca-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-z45nq" (UID: "8df4290a-39f3-4ea5-b30e-6ccd32e57bca") : secret "prometheus-operator-tls" not found
Sep 07 12:54:03.799392 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.799369    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8df4290a-39f3-4ea5-b30e-6ccd32e57bca-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-z45nq\" (UID: \"8df4290a-39f3-4ea5-b30e-6ccd32e57bca\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq"
Sep 07 12:54:03.801064 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.801033    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8df4290a-39f3-4ea5-b30e-6ccd32e57bca-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-z45nq\" (UID: \"8df4290a-39f3-4ea5-b30e-6ccd32e57bca\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq"
Sep 07 12:54:03.808282 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:03.808259    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48xlb\" (UniqueName: \"kubernetes.io/projected/8df4290a-39f3-4ea5-b30e-6ccd32e57bca-kube-api-access-48xlb\") pod \"prometheus-operator-6f4946db56-z45nq\" (UID: \"8df4290a-39f3-4ea5-b30e-6ccd32e57bca\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq"
Sep 07 12:54:04.302522 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:04.302481    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8df4290a-39f3-4ea5-b30e-6ccd32e57bca-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-z45nq\" (UID: \"8df4290a-39f3-4ea5-b30e-6ccd32e57bca\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq"
Sep 07 12:54:04.305062 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:04.305029    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8df4290a-39f3-4ea5-b30e-6ccd32e57bca-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-z45nq\" (UID: \"8df4290a-39f3-4ea5-b30e-6ccd32e57bca\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq"
Sep 07 12:54:04.565483 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:04.565382    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq"
Sep 07 12:54:04.836251 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:04.836172    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-z45nq"]
Sep 07 12:54:04.839356 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:54:04.839312    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df4290a_39f3_4ea5_b30e_6ccd32e57bca.slice/crio-845f3424a1d54b2ce8ce83fa6e3e7c250faa7ae930613f79d060f59f5e8b2536 WatchSource:0}: Error finding container 845f3424a1d54b2ce8ce83fa6e3e7c250faa7ae930613f79d060f59f5e8b2536: Status 404 returned error can't find the container with id 845f3424a1d54b2ce8ce83fa6e3e7c250faa7ae930613f79d060f59f5e8b2536
Sep 07 12:54:05.743281 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:05.743242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq" event={"ID":"8df4290a-39f3-4ea5-b30e-6ccd32e57bca","Type":"ContainerStarted","Data":"845f3424a1d54b2ce8ce83fa6e3e7c250faa7ae930613f79d060f59f5e8b2536"}
Sep 07 12:54:06.748300 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:06.748260    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq" event={"ID":"8df4290a-39f3-4ea5-b30e-6ccd32e57bca","Type":"ContainerStarted","Data":"01317bb5af9af6ccaa5e8ebf41c7965b3405675b95e224ca6382b9d6fae4aad5"}
Sep 07 12:54:06.748691 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:06.748378    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq" event={"ID":"8df4290a-39f3-4ea5-b30e-6ccd32e57bca","Type":"ContainerStarted","Data":"a277fba1f4a28d0eac8360723f43cda17861ed28d9378341af8c5e4a5234093b"}
Sep 07 12:54:06.768701 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:06.768637    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-z45nq" podStartSLOduration=2.5284793260000002 podStartE2EDuration="3.768623906s" totalImagesPullingTime="1.24014458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:54:06.767693129 +0000 UTC m=+63.798559460" watchObservedRunningTime="2026-09-07 12:54:06.768623906 +0000 UTC m=+63.799490216"
Sep 07 12:54:09.037797 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.037763    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"]
Sep 07 12:54:09.042225 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.042205    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"
Sep 07 12:54:09.045163 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.045136    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mlh8x\""
Sep 07 12:54:09.045306 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.045142    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 07 12:54:09.045522 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.045506    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 07 12:54:09.050021 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.049999    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"]
Sep 07 12:54:09.055199 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.055177    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"]
Sep 07 12:54:09.055369 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.055320    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.056676 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.056658    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-279tc"]
Sep 07 12:54:09.058087 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.058067    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 07 12:54:09.058181 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.058151    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5s7vz\""
Sep 07 12:54:09.058291 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.058264    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 07 12:54:09.058485 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.058466    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 07 12:54:09.060604 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.060586    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.064422 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.064405    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 07 12:54:09.065087 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.065070    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 07 12:54:09.065166 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.065075    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 07 12:54:09.065442 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.065423    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l5rwz\""
Sep 07 12:54:09.071389 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.071366    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"]
Sep 07 12:54:09.135112 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135074    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/090dc0b6-5c88-4c1a-9138-6011e61be2cd-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.135280 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/090dc0b6-5c88-4c1a-9138-6011e61be2cd-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.135280 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135144    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-accelerators-collector-config\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.135280 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs\") pod \"network-metrics-daemon-cbgw4\" (UID: \"4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4\") " pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:54:09.135280 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135195    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64a0a42b-c68c-4645-98af-30682c2b4a35-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-lznzr\" (UID: \"64a0a42b-c68c-4645-98af-30682c2b4a35\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"
Sep 07 12:54:09.135280 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135210    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a852057-5609-46ab-b910-af4397e8165a-sys\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.135280 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135224    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a852057-5609-46ab-b910-af4397e8165a-root\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.135280 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135242    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a852057-5609-46ab-b910-af4397e8165a-metrics-client-ca\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.135280 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135275    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-wtmp\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.135628 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135291    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64a0a42b-c68c-4645-98af-30682c2b4a35-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-lznzr\" (UID: \"64a0a42b-c68c-4645-98af-30682c2b4a35\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"
Sep 07 12:54:09.135628 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135347    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f62kv\" (UniqueName: \"kubernetes.io/projected/9a852057-5609-46ab-b910-af4397e8165a-kube-api-access-f62kv\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.135628 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135369    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbc45\" (UniqueName: \"kubernetes.io/projected/64a0a42b-c68c-4645-98af-30682c2b4a35-kube-api-access-vbc45\") pod \"openshift-state-metrics-657fd6c54-lznzr\" (UID: \"64a0a42b-c68c-4645-98af-30682c2b4a35\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"
Sep 07 12:54:09.135628 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135385    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/090dc0b6-5c88-4c1a-9138-6011e61be2cd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.135628 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-textfile\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.135628 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135474    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pczbc\" (UniqueName: \"kubernetes.io/projected/090dc0b6-5c88-4c1a-9138-6011e61be2cd-kube-api-access-pczbc\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.135628 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135493    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/090dc0b6-5c88-4c1a-9138-6011e61be2cd-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.135628 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-tls\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.135628 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/090dc0b6-5c88-4c1a-9138-6011e61be2cd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.135628 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135556    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/64a0a42b-c68c-4645-98af-30682c2b4a35-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-lznzr\" (UID: \"64a0a42b-c68c-4645-98af-30682c2b4a35\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"
Sep 07 12:54:09.135628 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.135571    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.138338 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.138309    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 12:54:09.148189 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.148168    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4-metrics-certs\") pod \"network-metrics-daemon-cbgw4\" (UID: \"4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4\") " pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:54:09.236215 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.236173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/090dc0b6-5c88-4c1a-9138-6011e61be2cd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.236215 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.236218    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/64a0a42b-c68c-4645-98af-30682c2b4a35-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-lznzr\" (UID: \"64a0a42b-c68c-4645-98af-30682c2b4a35\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"
Sep 07 12:54:09.236505 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.236238    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.236505 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.236364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/090dc0b6-5c88-4c1a-9138-6011e61be2cd-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.236505 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.236404    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret\") pod \"global-pull-secret-syncer-vjgrh\" (UID: \"b22d72c4-fb0e-4f78-9e7e-0550e6928616\") " pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:54:09.236505 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.236423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/090dc0b6-5c88-4c1a-9138-6011e61be2cd-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.236505 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.236450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-accelerators-collector-config\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.236505 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.236481    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64a0a42b-c68c-4645-98af-30682c2b4a35-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-lznzr\" (UID: \"64a0a42b-c68c-4645-98af-30682c2b4a35\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"
Sep 07 12:54:09.236505 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.236506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a852057-5609-46ab-b910-af4397e8165a-sys\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.236834 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.236527    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a852057-5609-46ab-b910-af4397e8165a-root\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.236834 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.236559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a852057-5609-46ab-b910-af4397e8165a-metrics-client-ca\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.236834 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.236595    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-wtmp\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.236834 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.236621    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64a0a42b-c68c-4645-98af-30682c2b4a35-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-lznzr\" (UID: \"64a0a42b-c68c-4645-98af-30682c2b4a35\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"
Sep 07 12:54:09.236834 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.236658    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f62kv\" (UniqueName: \"kubernetes.io/projected/9a852057-5609-46ab-b910-af4397e8165a-kube-api-access-f62kv\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.236834 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.236684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbc45\" (UniqueName: \"kubernetes.io/projected/64a0a42b-c68c-4645-98af-30682c2b4a35-kube-api-access-vbc45\") pod \"openshift-state-metrics-657fd6c54-lznzr\" (UID: \"64a0a42b-c68c-4645-98af-30682c2b4a35\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"
Sep 07 12:54:09.236834 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.236707    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/090dc0b6-5c88-4c1a-9138-6011e61be2cd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.237229 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.237208    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-wtmp\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.237348 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.237208    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a852057-5609-46ab-b910-af4397e8165a-root\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.237442 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.237245    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a852057-5609-46ab-b910-af4397e8165a-sys\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.237522 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.237352    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-accelerators-collector-config\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.237609 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.237381    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-textfile\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.237729 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.237712    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pczbc\" (UniqueName: \"kubernetes.io/projected/090dc0b6-5c88-4c1a-9138-6011e61be2cd-kube-api-access-pczbc\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.237867 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.237830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/090dc0b6-5c88-4c1a-9138-6011e61be2cd-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.237963 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.237940    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a852057-5609-46ab-b910-af4397e8165a-metrics-client-ca\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.238056 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.238039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-tls\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.238433 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.238124    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/090dc0b6-5c88-4c1a-9138-6011e61be2cd-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.238545 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.237485    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64a0a42b-c68c-4645-98af-30682c2b4a35-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-lznzr\" (UID: \"64a0a42b-c68c-4645-98af-30682c2b4a35\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"
Sep 07 12:54:09.238650 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.237913    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/090dc0b6-5c88-4c1a-9138-6011e61be2cd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.238848 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:54:09.238831    2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 07 12:54:09.239143 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:54:09.239126    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-tls podName:9a852057-5609-46ab-b910-af4397e8165a nodeName:}" failed. No retries permitted until 2026-09-07 12:54:09.739105446 +0000 UTC m=+66.769971749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-tls") pod "node-exporter-279tc" (UID: "9a852057-5609-46ab-b910-af4397e8165a") : secret "node-exporter-tls" not found
Sep 07 12:54:09.239307 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.239025    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-textfile\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.239452 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.238923    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/090dc0b6-5c88-4c1a-9138-6011e61be2cd-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.239556 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.239303    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 12:54:09.239664 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.239576    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.239664 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.239611    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/090dc0b6-5c88-4c1a-9138-6011e61be2cd-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.239762 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.239688    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/090dc0b6-5c88-4c1a-9138-6011e61be2cd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.239840 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.239821    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/64a0a42b-c68c-4645-98af-30682c2b4a35-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-lznzr\" (UID: \"64a0a42b-c68c-4645-98af-30682c2b4a35\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"
Sep 07 12:54:09.240101 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.240086    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64a0a42b-c68c-4645-98af-30682c2b4a35-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-lznzr\" (UID: \"64a0a42b-c68c-4645-98af-30682c2b4a35\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"
Sep 07 12:54:09.247805 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.247780    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f62kv\" (UniqueName: \"kubernetes.io/projected/9a852057-5609-46ab-b910-af4397e8165a-kube-api-access-f62kv\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.248836 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.248817    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b22d72c4-fb0e-4f78-9e7e-0550e6928616-original-pull-secret\") pod \"global-pull-secret-syncer-vjgrh\" (UID: \"b22d72c4-fb0e-4f78-9e7e-0550e6928616\") " pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:54:09.249563 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.249541    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbc45\" (UniqueName: \"kubernetes.io/projected/64a0a42b-c68c-4645-98af-30682c2b4a35-kube-api-access-vbc45\") pod \"openshift-state-metrics-657fd6c54-lznzr\" (UID: \"64a0a42b-c68c-4645-98af-30682c2b4a35\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"
Sep 07 12:54:09.253781 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.253759    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pczbc\" (UniqueName: \"kubernetes.io/projected/090dc0b6-5c88-4c1a-9138-6011e61be2cd-kube-api-access-pczbc\") pod \"kube-state-metrics-74cb57df7-q7hxn\" (UID: \"090dc0b6-5c88-4c1a-9138-6011e61be2cd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.294622 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.294587    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gnfpv\""
Sep 07 12:54:09.302904 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.302881    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cbgw4"
Sep 07 12:54:09.338825 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.338789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvmx\" (UniqueName: \"kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx\") pod \"network-check-target-5tbdn\" (UID: \"fbbea119-b216-4988-ba0d-7c408dac7fc6\") " pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:54:09.341430 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.341404    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 12:54:09.350575 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.350553    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"
Sep 07 12:54:09.352593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.352567    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 12:54:09.363816 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.363790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zvmx\" (UniqueName: \"kubernetes.io/projected/fbbea119-b216-4988-ba0d-7c408dac7fc6-kube-api-access-5zvmx\") pod \"network-check-target-5tbdn\" (UID: \"fbbea119-b216-4988-ba0d-7c408dac7fc6\") " pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:54:09.365689 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.365493    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"
Sep 07 12:54:09.403483 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.402469    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xfxxc\""
Sep 07 12:54:09.403483 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.402858    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjgrh"
Sep 07 12:54:09.407641 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.407550    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:54:09.575115 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.575027    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cbgw4"]
Sep 07 12:54:09.577505 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:54:09.577477    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a40ecf1_4b6f_4a0f_bbee_8bfa92fa4cd4.slice/crio-d24bebd5f011a7f8a865ecc2b428657eed69051e11ded53134a69add48e02e66 WatchSource:0}: Error finding container d24bebd5f011a7f8a865ecc2b428657eed69051e11ded53134a69add48e02e66: Status 404 returned error can't find the container with id d24bebd5f011a7f8a865ecc2b428657eed69051e11ded53134a69add48e02e66
Sep 07 12:54:09.635971 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.635937    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr"]
Sep 07 12:54:09.639003 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:54:09.638979    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a0a42b_c68c_4645_98af_30682c2b4a35.slice/crio-d2b868dad3e89b836b796962f84b3e1917966b48ed25391a66300395a1d3376c WatchSource:0}: Error finding container d2b868dad3e89b836b796962f84b3e1917966b48ed25391a66300395a1d3376c: Status 404 returned error can't find the container with id d2b868dad3e89b836b796962f84b3e1917966b48ed25391a66300395a1d3376c
Sep 07 12:54:09.657893 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.657867    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn"]
Sep 07 12:54:09.673266 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:54:09.673240    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090dc0b6_5c88_4c1a_9138_6011e61be2cd.slice/crio-c10914e0ebdcda5ba5636a13920d8254d78dbcef1ff64aab332760180c3c003b WatchSource:0}: Error finding container c10914e0ebdcda5ba5636a13920d8254d78dbcef1ff64aab332760180c3c003b: Status 404 returned error can't find the container with id c10914e0ebdcda5ba5636a13920d8254d78dbcef1ff64aab332760180c3c003b
Sep 07 12:54:09.704549 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.704509    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5tbdn"]
Sep 07 12:54:09.705444 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.705424    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vjgrh"]
Sep 07 12:54:09.707687 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:54:09.707663    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbea119_b216_4988_ba0d_7c408dac7fc6.slice/crio-38d47f06b44c87962603892a8e9edf2d17922edc373158f6b2122ac226a4ab8b WatchSource:0}: Error finding container 38d47f06b44c87962603892a8e9edf2d17922edc373158f6b2122ac226a4ab8b: Status 404 returned error can't find the container with id 38d47f06b44c87962603892a8e9edf2d17922edc373158f6b2122ac226a4ab8b
Sep 07 12:54:09.710754 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:54:09.710735    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22d72c4_fb0e_4f78_9e7e_0550e6928616.slice/crio-cbca92f88bcb322c5a23179437e9a5667ecf082c56e61abe3c2470856ee137c0 WatchSource:0}: Error finding container cbca92f88bcb322c5a23179437e9a5667ecf082c56e61abe3c2470856ee137c0: Status 404 returned error can't find the container with id cbca92f88bcb322c5a23179437e9a5667ecf082c56e61abe3c2470856ee137c0
Sep 07 12:54:09.726625 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.726603    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sb8hw"
Sep 07 12:54:09.742634 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.742281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-tls\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.745149 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.744985    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a852057-5609-46ab-b910-af4397e8165a-node-exporter-tls\") pod \"node-exporter-279tc\" (UID: \"9a852057-5609-46ab-b910-af4397e8165a\") " pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.762938 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.759658    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn" event={"ID":"090dc0b6-5c88-4c1a-9138-6011e61be2cd","Type":"ContainerStarted","Data":"c10914e0ebdcda5ba5636a13920d8254d78dbcef1ff64aab332760180c3c003b"}
Sep 07 12:54:09.762938 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.762301    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr" event={"ID":"64a0a42b-c68c-4645-98af-30682c2b4a35","Type":"ContainerStarted","Data":"7e6eec7f1902f76e86def10aba2f449f083b9627cb387bb1a06a4f209eb3245d"}
Sep 07 12:54:09.762938 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.762385    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr" event={"ID":"64a0a42b-c68c-4645-98af-30682c2b4a35","Type":"ContainerStarted","Data":"d2b868dad3e89b836b796962f84b3e1917966b48ed25391a66300395a1d3376c"}
Sep 07 12:54:09.763780 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.763754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vjgrh" event={"ID":"b22d72c4-fb0e-4f78-9e7e-0550e6928616","Type":"ContainerStarted","Data":"cbca92f88bcb322c5a23179437e9a5667ecf082c56e61abe3c2470856ee137c0"}
Sep 07 12:54:09.765207 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.765183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cbgw4" event={"ID":"4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4","Type":"ContainerStarted","Data":"d24bebd5f011a7f8a865ecc2b428657eed69051e11ded53134a69add48e02e66"}
Sep 07 12:54:09.766423 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.766404    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5tbdn" event={"ID":"fbbea119-b216-4988-ba0d-7c408dac7fc6","Type":"ContainerStarted","Data":"38d47f06b44c87962603892a8e9edf2d17922edc373158f6b2122ac226a4ab8b"}
Sep 07 12:54:09.970394 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:09.970281    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-279tc"
Sep 07 12:54:09.980506 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:54:09.980480    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a852057_5609_46ab_b910_af4397e8165a.slice/crio-a983eee8d8ff42ec3393b0b6abeb515497549e8365e6b9a7816a6cd2ea095dd6 WatchSource:0}: Error finding container a983eee8d8ff42ec3393b0b6abeb515497549e8365e6b9a7816a6cd2ea095dd6: Status 404 returned error can't find the container with id a983eee8d8ff42ec3393b0b6abeb515497549e8365e6b9a7816a6cd2ea095dd6
Sep 07 12:54:10.153798 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.153746    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 12:54:10.157973 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.157951    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.160509 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.160488    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4kk8g\""
Sep 07 12:54:10.160867 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.160849    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 07 12:54:10.161561 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.161537    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 07 12:54:10.161668 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.161562    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 07 12:54:10.161668 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.161609    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 07 12:54:10.161668 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.161564    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 07 12:54:10.161668 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.161659    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 07 12:54:10.162045 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.161611    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 07 12:54:10.162045 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.161565    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 07 12:54:10.162045 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.161852    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 07 12:54:10.175896 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.175799    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 12:54:10.246738 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.246643    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.246738 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.246697    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.246738 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.246733    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2p4\" (UniqueName: \"kubernetes.io/projected/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-kube-api-access-6c2p4\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.247021 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.246757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.247021 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.246784    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-config-out\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.247021 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.246809    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-config-volume\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.247021 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.246871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.247021 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.246922    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.247021 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.246954    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.247021 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.246983    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.247021 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.247009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-web-config\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.247285 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.247093    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.247285 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.247134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.348616 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.348528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.349346 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.349290    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.349477 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.349366    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.349477 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.349394    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-web-config\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.349477 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.349449    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.349579 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.349478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.349579 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.349538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.349651 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.349576    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.349651 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.349610    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2p4\" (UniqueName: \"kubernetes.io/projected/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-kube-api-access-6c2p4\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.349651 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.349636    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.350075 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.349660    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-config-out\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.350075 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.349683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-config-volume\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.350075 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.349715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.351070 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.350400    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.351434 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.351411    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.352720 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.352695    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.360566 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.360483    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-config-volume\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.368043 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.368013    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2p4\" (UniqueName: \"kubernetes.io/projected/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-kube-api-access-6c2p4\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.370271 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.370168    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.370271 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.370212    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.370538 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.370516    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-config-out\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.370660 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.370638    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.370951 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.370928    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.371095 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.371044    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-web-config\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.371446 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.371423    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.373868 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.373837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.490776 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.490507    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:54:10.777623 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.777512    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr" event={"ID":"64a0a42b-c68c-4645-98af-30682c2b4a35","Type":"ContainerStarted","Data":"3727664986b58fa2fe378c42b1e1e8fef2aa5ac3cc0fe7f6484f624bd241cec0"}
Sep 07 12:54:10.780137 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:10.780056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-279tc" event={"ID":"9a852057-5609-46ab-b910-af4397e8165a","Type":"ContainerStarted","Data":"a983eee8d8ff42ec3393b0b6abeb515497549e8365e6b9a7816a6cd2ea095dd6"}
Sep 07 12:54:11.053832 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.051947    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5874f5666d-jdf7t"]
Sep 07 12:54:11.056149 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.056121    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.060502 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.060475    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 07 12:54:11.060635 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.060514    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 07 12:54:11.060635 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.060574    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 07 12:54:11.060635 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.060620    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gjcjl\""
Sep 07 12:54:11.060792 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.060692    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 07 12:54:11.060853 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.060825    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 07 12:54:11.060853 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.060783    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2bli25quf1a3h\""
Sep 07 12:54:11.068969 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.068927    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5874f5666d-jdf7t"]
Sep 07 12:54:11.157550 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.157507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-thanos-querier-tls\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.158048 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.157563    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.158048 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.157613    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc908e22-0fe5-4665-abae-f5e5197585d7-metrics-client-ca\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.158048 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.157648    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f7w4\" (UniqueName: \"kubernetes.io/projected/fc908e22-0fe5-4665-abae-f5e5197585d7-kube-api-access-2f7w4\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.158048 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.157685    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.158048 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.157712    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-grpc-tls\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.158048 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.157742    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.158048 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.157771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.258850 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.258814    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc908e22-0fe5-4665-abae-f5e5197585d7-metrics-client-ca\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.259014 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.258876    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f7w4\" (UniqueName: \"kubernetes.io/projected/fc908e22-0fe5-4665-abae-f5e5197585d7-kube-api-access-2f7w4\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.259014 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.258915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.259014 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.258941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-grpc-tls\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.259014 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.258972    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.259014 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.259001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.259224 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.259060    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-thanos-querier-tls\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.259224 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.259091    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.259792 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.259674    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc908e22-0fe5-4665-abae-f5e5197585d7-metrics-client-ca\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.262393 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.262321    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.263032 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.263004    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.263963 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.263939    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-grpc-tls\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.264102 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.264080    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-thanos-querier-tls\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.264373 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.264322    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.265432 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.265405    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc908e22-0fe5-4665-abae-f5e5197585d7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.302404 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.302371    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f7w4\" (UniqueName: \"kubernetes.io/projected/fc908e22-0fe5-4665-abae-f5e5197585d7-kube-api-access-2f7w4\") pod \"thanos-querier-5874f5666d-jdf7t\" (UID: \"fc908e22-0fe5-4665-abae-f5e5197585d7\") " pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:11.372549 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:11.372468    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:12.054933 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:12.054903    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5874f5666d-jdf7t"]
Sep 07 12:54:12.056952 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:12.056864    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 12:54:12.117493 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:54:12.117459    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc908e22_0fe5_4665_abae_f5e5197585d7.slice/crio-e0828d7779fe3417eb4f009e2b4d66ce43297e5effe6a3538b88055d2224a9c7 WatchSource:0}: Error finding container e0828d7779fe3417eb4f009e2b4d66ce43297e5effe6a3538b88055d2224a9c7: Status 404 returned error can't find the container with id e0828d7779fe3417eb4f009e2b4d66ce43297e5effe6a3538b88055d2224a9c7
Sep 07 12:54:12.118446 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:54:12.118366    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25fea9ce_cdf0_4b1c_86a9_8aa64075ea16.slice/crio-28490d8ec337c0c52c1fc95871a4e43fd8100b36967e2ad30cb81e077474deb2 WatchSource:0}: Error finding container 28490d8ec337c0c52c1fc95871a4e43fd8100b36967e2ad30cb81e077474deb2: Status 404 returned error can't find the container with id 28490d8ec337c0c52c1fc95871a4e43fd8100b36967e2ad30cb81e077474deb2
Sep 07 12:54:12.792393 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:12.792289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cbgw4" event={"ID":"4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4","Type":"ContainerStarted","Data":"da80a0d8dfa994cb755a887227f4f8d1fb5bd3f3108782adfbfbd111d536a480"}
Sep 07 12:54:12.792393 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:12.792387    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cbgw4" event={"ID":"4a40ecf1-4b6f-4a0f-bbee-8bfa92fa4cd4","Type":"ContainerStarted","Data":"fcc55daee07b54a9bdb1312760e41af89604976d734c7d5d2ecce2b363b6c575"}
Sep 07 12:54:12.796845 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:12.796019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn" event={"ID":"090dc0b6-5c88-4c1a-9138-6011e61be2cd","Type":"ContainerStarted","Data":"327b1ef2d9b57c93c0b0ee14cbd7b88830ef084a75da2dc4788933ba77974eb1"}
Sep 07 12:54:12.796845 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:12.796079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn" event={"ID":"090dc0b6-5c88-4c1a-9138-6011e61be2cd","Type":"ContainerStarted","Data":"2376ca190a1a722996a253111c5441fc3107b83f8d41a779363d20fa3eadd7fa"}
Sep 07 12:54:12.796845 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:12.796110    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn" event={"ID":"090dc0b6-5c88-4c1a-9138-6011e61be2cd","Type":"ContainerStarted","Data":"8da33cda7c4a5ec01a293beb2fa8a3be894355217e267a5ccf07b8f65fd89343"}
Sep 07 12:54:12.800124 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:12.799649    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr" event={"ID":"64a0a42b-c68c-4645-98af-30682c2b4a35","Type":"ContainerStarted","Data":"a0ff357fe20bc304e915938c4ab3922e18086e1253f29146e5973971b5092867"}
Sep 07 12:54:12.802005 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:12.801977    2565 generic.go:356] "Generic (PLEG): container finished" podID="9a852057-5609-46ab-b910-af4397e8165a" containerID="c28b7cd5814f56f38c24eb9005c9e7bf39777d31d8f39666df6635e1adfab2c4" exitCode=0
Sep 07 12:54:12.802516 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:12.802494    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-279tc" event={"ID":"9a852057-5609-46ab-b910-af4397e8165a","Type":"ContainerDied","Data":"c28b7cd5814f56f38c24eb9005c9e7bf39777d31d8f39666df6635e1adfab2c4"}
Sep 07 12:54:12.804184 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:12.804071    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t" event={"ID":"fc908e22-0fe5-4665-abae-f5e5197585d7","Type":"ContainerStarted","Data":"e0828d7779fe3417eb4f009e2b4d66ce43297e5effe6a3538b88055d2224a9c7"}
Sep 07 12:54:12.805769 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:12.805743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16","Type":"ContainerStarted","Data":"28490d8ec337c0c52c1fc95871a4e43fd8100b36967e2ad30cb81e077474deb2"}
Sep 07 12:54:12.836463 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:12.836358    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cbgw4" podStartSLOduration=67.657599744 podStartE2EDuration="1m9.8363186s" totalImagesPullingTime="2.178718856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:54:12.812579304 +0000 UTC m=+69.843445615" watchObservedRunningTime="2026-09-07 12:54:12.8363186 +0000 UTC m=+69.867184912"
Sep 07 12:54:12.859342 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:12.859266    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-lznzr" podStartSLOduration=1.869025808 podStartE2EDuration="3.859246707s" totalImagesPullingTime="1.990220899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:54:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:54:12.857724399 +0000 UTC m=+69.888590709" watchObservedRunningTime="2026-09-07 12:54:12.859246707 +0000 UTC m=+69.890113018"
Sep 07 12:54:13.497134 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.497065    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-q7hxn" podStartSLOduration=2.041485681 podStartE2EDuration="4.497041674s" totalImagesPullingTime="2.455555993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:54:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:54:12.886591352 +0000 UTC m=+69.917457658" watchObservedRunningTime="2026-09-07 12:54:13.497041674 +0000 UTC m=+70.527907984"
Sep 07 12:54:13.498017 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.497989    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-567854bfc5-m9hbl"]
Sep 07 12:54:13.500796 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.500770    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.503216 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.503195    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 07 12:54:13.504172 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.504152    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hwjft\""
Sep 07 12:54:13.504369 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.504351    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 07 12:54:13.504558 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.504544    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-12rg1vcpc64ok\""
Sep 07 12:54:13.504639 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.504619    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 07 12:54:13.504810 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.504788    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 07 12:54:13.512620 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.512592    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-567854bfc5-m9hbl"]
Sep 07 12:54:13.583366 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.583301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnfr2\" (UniqueName: \"kubernetes.io/projected/9ef7c14b-87f2-4057-aee8-b32c36d0535a-kube-api-access-jnfr2\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.583514 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.583383    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9ef7c14b-87f2-4057-aee8-b32c36d0535a-client-ca-bundle\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.583514 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.583419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9ef7c14b-87f2-4057-aee8-b32c36d0535a-metrics-server-audit-profiles\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.583514 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.583461    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9ef7c14b-87f2-4057-aee8-b32c36d0535a-secret-metrics-server-tls\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.583514 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.583488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9ef7c14b-87f2-4057-aee8-b32c36d0535a-secret-metrics-server-client-certs\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.583687 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.583519    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9ef7c14b-87f2-4057-aee8-b32c36d0535a-audit-log\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.583687 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.583573    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef7c14b-87f2-4057-aee8-b32c36d0535a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.684594 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.684558    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9ef7c14b-87f2-4057-aee8-b32c36d0535a-audit-log\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.684779 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.684604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef7c14b-87f2-4057-aee8-b32c36d0535a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.684779 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.684670    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnfr2\" (UniqueName: \"kubernetes.io/projected/9ef7c14b-87f2-4057-aee8-b32c36d0535a-kube-api-access-jnfr2\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.684779 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.684692    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9ef7c14b-87f2-4057-aee8-b32c36d0535a-client-ca-bundle\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.684779 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.684722    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9ef7c14b-87f2-4057-aee8-b32c36d0535a-metrics-server-audit-profiles\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.684779 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.684762    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9ef7c14b-87f2-4057-aee8-b32c36d0535a-secret-metrics-server-tls\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.685031 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.684785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9ef7c14b-87f2-4057-aee8-b32c36d0535a-secret-metrics-server-client-certs\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.685081 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.685042    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9ef7c14b-87f2-4057-aee8-b32c36d0535a-audit-log\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.685455 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.685424    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef7c14b-87f2-4057-aee8-b32c36d0535a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.685811 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.685760    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9ef7c14b-87f2-4057-aee8-b32c36d0535a-metrics-server-audit-profiles\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.687630 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.687609    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9ef7c14b-87f2-4057-aee8-b32c36d0535a-client-ca-bundle\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.688097 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.688054    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9ef7c14b-87f2-4057-aee8-b32c36d0535a-secret-metrics-server-client-certs\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.688097 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.688053    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9ef7c14b-87f2-4057-aee8-b32c36d0535a-secret-metrics-server-tls\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.695551 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.695533    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnfr2\" (UniqueName: \"kubernetes.io/projected/9ef7c14b-87f2-4057-aee8-b32c36d0535a-kube-api-access-jnfr2\") pod \"metrics-server-567854bfc5-m9hbl\" (UID: \"9ef7c14b-87f2-4057-aee8-b32c36d0535a\") " pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.812949 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.812918    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-5sp6l"]
Sep 07 12:54:13.814702 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.814673    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:13.815111 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.815089    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5sp6l"
Sep 07 12:54:13.817532 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.817508    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 07 12:54:13.817636 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.817518    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v8dwh\""
Sep 07 12:54:13.826064 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.826040    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-5sp6l"]
Sep 07 12:54:13.886259 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.886223    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4b9r\" (UniqueName: \"kubernetes.io/projected/c32174db-fc07-4988-9eb2-19b2908afa4b-kube-api-access-w4b9r\") pod \"monitoring-plugin-85bbbbbf4-5sp6l\" (UID: \"c32174db-fc07-4988-9eb2-19b2908afa4b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5sp6l"
Sep 07 12:54:13.886457 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.886295    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c32174db-fc07-4988-9eb2-19b2908afa4b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-5sp6l\" (UID: \"c32174db-fc07-4988-9eb2-19b2908afa4b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5sp6l"
Sep 07 12:54:13.987470 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.987436    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4b9r\" (UniqueName: \"kubernetes.io/projected/c32174db-fc07-4988-9eb2-19b2908afa4b-kube-api-access-w4b9r\") pod \"monitoring-plugin-85bbbbbf4-5sp6l\" (UID: \"c32174db-fc07-4988-9eb2-19b2908afa4b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5sp6l"
Sep 07 12:54:13.987470 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.987472    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c32174db-fc07-4988-9eb2-19b2908afa4b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-5sp6l\" (UID: \"c32174db-fc07-4988-9eb2-19b2908afa4b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5sp6l"
Sep 07 12:54:13.990198 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:13.990165    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c32174db-fc07-4988-9eb2-19b2908afa4b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-5sp6l\" (UID: \"c32174db-fc07-4988-9eb2-19b2908afa4b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5sp6l"
Sep 07 12:54:14.000516 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:14.000483    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4b9r\" (UniqueName: \"kubernetes.io/projected/c32174db-fc07-4988-9eb2-19b2908afa4b-kube-api-access-w4b9r\") pod \"monitoring-plugin-85bbbbbf4-5sp6l\" (UID: \"c32174db-fc07-4988-9eb2-19b2908afa4b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5sp6l"
Sep 07 12:54:14.127201 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:14.127108    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5sp6l"
Sep 07 12:54:15.314112 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.314075    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 12:54:15.324595 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.324554    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.337031 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.333959    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d1lvn1p2c7847\""
Sep 07 12:54:15.337031 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.334342    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 07 12:54:15.337031 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.334598    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 07 12:54:15.337031 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.334874    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 07 12:54:15.337031 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.335118    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 07 12:54:15.337031 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.335219    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 07 12:54:15.337031 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.335288    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 07 12:54:15.337031 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.335572    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 07 12:54:15.337031 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.335871    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 07 12:54:15.337031 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.336078    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 07 12:54:15.337031 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.336292    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 07 12:54:15.337031 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.336619    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 07 12:54:15.337031 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.336845    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 07 12:54:15.337791 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.337094    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 07 12:54:15.337791 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.337626    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xdtxc\""
Sep 07 12:54:15.341658 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.341542    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 12:54:15.342035 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.341973    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 07 12:54:15.399587 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.399539    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.399773 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.399620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.399773 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.399646    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.399773 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.399675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.399773 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.399702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqllb\" (UniqueName: \"kubernetes.io/projected/ece4bc9d-7296-429a-9029-91b9f6c86689-kube-api-access-tqllb\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.399773 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.399737    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.399773 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.399754    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.400110 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.399787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.400110 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.399818    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.400110 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.399843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-config\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.400110 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.399862    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.400110 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.399926    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ece4bc9d-7296-429a-9029-91b9f6c86689-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.400110 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.399968    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.400110 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.400014    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.400110 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.400041    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.400110 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.400074    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.400110 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.400101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ece4bc9d-7296-429a-9029-91b9f6c86689-config-out\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.400646 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.400123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-web-config\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.400646 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.400155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.400646 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.400267    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.501491 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-config\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.501680 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.501680 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ece4bc9d-7296-429a-9029-91b9f6c86689-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.501680 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.501680 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.501680 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501629    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.501680 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501661    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.502000 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501686    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ece4bc9d-7296-429a-9029-91b9f6c86689-config-out\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.502000 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501709    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-web-config\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.502000 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.502000 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501774    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.502000 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.502000 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.502000 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501878    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.502000 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501911    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.502000 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501937    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqllb\" (UniqueName: \"kubernetes.io/projected/ece4bc9d-7296-429a-9029-91b9f6c86689-kube-api-access-tqllb\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.502000 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501964    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.502000 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.501988    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.502477 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.502023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.502477 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.502049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.504005 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.503972    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.504871 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.504827    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.504975 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.504843    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.505134 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.505071    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.506117 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.505356    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.506117 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.505637    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.506117 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.505800    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.506117 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.505861    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.508367 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.508118    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-config\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.509705 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.509657    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.511199 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.510390    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.511199 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.510718    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ece4bc9d-7296-429a-9029-91b9f6c86689-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.511199 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.510763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ece4bc9d-7296-429a-9029-91b9f6c86689-config-out\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.511199 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.511155    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.511839 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.511355    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-web-config\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.511839 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.511488    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.512733 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.512711    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.514140 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.511087    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.514971 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.514909    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqllb\" (UniqueName: \"kubernetes.io/projected/ece4bc9d-7296-429a-9029-91b9f6c86689-kube-api-access-tqllb\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.515606 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.515582    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:15.653435 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:15.653351    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:16.730077 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.730045    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-567854bfc5-m9hbl"]
Sep 07 12:54:16.744890 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.744374    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 12:54:16.745743 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.745634    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-5sp6l"]
Sep 07 12:54:16.747973 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:54:16.747943    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece4bc9d_7296_429a_9029_91b9f6c86689.slice/crio-454f4e37e5c42b14cd76e4dbc253042a9380f8f965121f9cc4535c7fc7c09241 WatchSource:0}: Error finding container 454f4e37e5c42b14cd76e4dbc253042a9380f8f965121f9cc4535c7fc7c09241: Status 404 returned error can't find the container with id 454f4e37e5c42b14cd76e4dbc253042a9380f8f965121f9cc4535c7fc7c09241
Sep 07 12:54:16.749029 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:54:16.748992    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32174db_fc07_4988_9eb2_19b2908afa4b.slice/crio-50f56050b29bd4b2def1581fd26852332dd90d18dc9fbcaa5521ec8e85cc3f36 WatchSource:0}: Error finding container 50f56050b29bd4b2def1581fd26852332dd90d18dc9fbcaa5521ec8e85cc3f36: Status 404 returned error can't find the container with id 50f56050b29bd4b2def1581fd26852332dd90d18dc9fbcaa5521ec8e85cc3f36
Sep 07 12:54:16.825971 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.825937    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl" event={"ID":"9ef7c14b-87f2-4057-aee8-b32c36d0535a","Type":"ContainerStarted","Data":"e726cdd2c26014fd89ccc753ece7476affefee25a052d5d2ffdf082180fbd420"}
Sep 07 12:54:16.828585 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.828523    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vjgrh" event={"ID":"b22d72c4-fb0e-4f78-9e7e-0550e6928616","Type":"ContainerStarted","Data":"2c1942f769b766222c6f6b5dd3871c71872bd7a873986d3a9c9f744fa5563098"}
Sep 07 12:54:16.831074 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.830668    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ece4bc9d-7296-429a-9029-91b9f6c86689","Type":"ContainerStarted","Data":"a331e53fc20132115118c5dec516fb946fb0dfebac146cafb062eb744b1372df"}
Sep 07 12:54:16.831074 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.830744    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ece4bc9d-7296-429a-9029-91b9f6c86689","Type":"ContainerStarted","Data":"454f4e37e5c42b14cd76e4dbc253042a9380f8f965121f9cc4535c7fc7c09241"}
Sep 07 12:54:16.832412 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.832357    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5tbdn" event={"ID":"fbbea119-b216-4988-ba0d-7c408dac7fc6","Type":"ContainerStarted","Data":"b4948b5aeb4fbc06e9918a811426c2914c7a46da1b8e49f21a3446b71df8dcb1"}
Sep 07 12:54:16.832520 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.832420    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:54:16.836308 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.836278    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-279tc" event={"ID":"9a852057-5609-46ab-b910-af4397e8165a","Type":"ContainerStarted","Data":"f832106b2cdaf909bb2e8b26056b6757e76096f8bf27e7d306106be08ba4fc05"}
Sep 07 12:54:16.836444 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.836351    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-279tc" event={"ID":"9a852057-5609-46ab-b910-af4397e8165a","Type":"ContainerStarted","Data":"12b76a8d85e5b1e8c678a40e0bc8d0496fe6cf6f7b5584f1e336b3907727edec"}
Sep 07 12:54:16.838937 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.838916    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t" event={"ID":"fc908e22-0fe5-4665-abae-f5e5197585d7","Type":"ContainerStarted","Data":"0f57b3b4e3673212bf392ce9fcefcd086aad4f981cbf21d7fb53e144971310b6"}
Sep 07 12:54:16.839039 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.838967    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t" event={"ID":"fc908e22-0fe5-4665-abae-f5e5197585d7","Type":"ContainerStarted","Data":"3a25c9231129478ae86fce63212bd708d1974186ddfff08ee73e2e1b6b489f13"}
Sep 07 12:54:16.839039 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.839000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t" event={"ID":"fc908e22-0fe5-4665-abae-f5e5197585d7","Type":"ContainerStarted","Data":"e0cc9c407d73bdde85949c86c38d64794e6a2a909500ffae09675b8609d6cbd4"}
Sep 07 12:54:16.840534 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.840507    2565 generic.go:356] "Generic (PLEG): container finished" podID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerID="9f778aeaf6351c863504ca3a94effe99f0bd8b1e1ee26ba9c6aeff8c77281ce5" exitCode=0
Sep 07 12:54:16.840622 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.840581    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16","Type":"ContainerDied","Data":"9f778aeaf6351c863504ca3a94effe99f0bd8b1e1ee26ba9c6aeff8c77281ce5"}
Sep 07 12:54:16.843069 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.843047    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5sp6l" event={"ID":"c32174db-fc07-4988-9eb2-19b2908afa4b","Type":"ContainerStarted","Data":"50f56050b29bd4b2def1581fd26852332dd90d18dc9fbcaa5521ec8e85cc3f36"}
Sep 07 12:54:16.851515 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.851469    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vjgrh" podStartSLOduration=67.126666579 podStartE2EDuration="1m13.851454012s" totalImagesPullingTime="6.724787433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:54:16.850520873 +0000 UTC m=+73.881387186" watchObservedRunningTime="2026-09-07 12:54:16.851454012 +0000 UTC m=+73.882320316"
Sep 07 12:54:16.871974 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.871915    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5tbdn" podStartSLOduration=67.150823966 podStartE2EDuration="1m13.871898698s" totalImagesPullingTime="6.721074732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:54:16.870622313 +0000 UTC m=+73.901488622" watchObservedRunningTime="2026-09-07 12:54:16.871898698 +0000 UTC m=+73.902765023"
Sep 07 12:54:16.897109 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:16.897047    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-279tc" podStartSLOduration=5.552265663 podStartE2EDuration="7.897028072s" totalImagesPullingTime="2.210048808s" totalInitContainerRuntime="134.713601ms" isStatefulPod=true podCreationTimestamp="2026-09-07 12:54:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:54:16.895294574 +0000 UTC m=+73.926160897" watchObservedRunningTime="2026-09-07 12:54:16.897028072 +0000 UTC m=+73.927894382"
Sep 07 12:54:17.850667 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:17.850308    2565 generic.go:356] "Generic (PLEG): container finished" podID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerID="a331e53fc20132115118c5dec516fb946fb0dfebac146cafb062eb744b1372df" exitCode=0
Sep 07 12:54:17.850667 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:17.850365    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ece4bc9d-7296-429a-9029-91b9f6c86689","Type":"ContainerDied","Data":"a331e53fc20132115118c5dec516fb946fb0dfebac146cafb062eb744b1372df"}
Sep 07 12:54:19.864583 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.864542    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t" event={"ID":"fc908e22-0fe5-4665-abae-f5e5197585d7","Type":"ContainerStarted","Data":"c7a58a272dccd564f2d2296a897d0a7cc22c238d9b588b00a878b3f60ea7f4be"}
Sep 07 12:54:19.865023 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.864628    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t" event={"ID":"fc908e22-0fe5-4665-abae-f5e5197585d7","Type":"ContainerStarted","Data":"f53754b4c5a4d5ae68a15834d91a03dc93f15ca6525ac74c32cb37de1b5351bc"}
Sep 07 12:54:19.865023 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.864657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t" event={"ID":"fc908e22-0fe5-4665-abae-f5e5197585d7","Type":"ContainerStarted","Data":"7618ce762ea62d63950b0e63b2828e00f608ada8e039b1243c0dc844f706d98a"}
Sep 07 12:54:19.865023 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.864703    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:19.868277 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.868253    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16","Type":"ContainerStarted","Data":"6d135997dfebff46b66d09746ae2005ce52f7a49f42f603ef4150906e7457e0d"}
Sep 07 12:54:19.868431 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.868317    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16","Type":"ContainerStarted","Data":"31b9f0cc0ddb554d4bb98c177db6578c2769d6407aa01d05d4d846ba5acb0d9b"}
Sep 07 12:54:19.868431 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.868378    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16","Type":"ContainerStarted","Data":"9282c5ddd6b70fade474fe506b38e94ed1777d6ef3338ca2a4aeb623dbadde4a"}
Sep 07 12:54:19.868431 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.868410    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16","Type":"ContainerStarted","Data":"48a333867d35821d21fa0d742affd25b167a0caedc184ec1bf172ec90f22f52b"}
Sep 07 12:54:19.868575 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.868449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16","Type":"ContainerStarted","Data":"3d13232ff07457f0c63ef5b3261de83022349ffeac3c167d2da2a1a611f5b6a3"}
Sep 07 12:54:19.868575 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.868496    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16","Type":"ContainerStarted","Data":"8e131ba9eceb43ab0a61b4e3f331dab148e5577360d90b3d7f37e2f71f9c602d"}
Sep 07 12:54:19.869974 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.869943    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5sp6l" event={"ID":"c32174db-fc07-4988-9eb2-19b2908afa4b","Type":"ContainerStarted","Data":"d850ad5edb04703b59c809a98a7525d48d30c6d78bef0b9eb3209138eecd37ad"}
Sep 07 12:54:19.870082 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.869995    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5sp6l"
Sep 07 12:54:19.871604 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.871581    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl" event={"ID":"9ef7c14b-87f2-4057-aee8-b32c36d0535a","Type":"ContainerStarted","Data":"93036dbea6ee8845b36f4f6aff52cfbc3267d360cae30dba1160fec3004ad653"}
Sep 07 12:54:19.875422 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.875402    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5sp6l"
Sep 07 12:54:19.889075 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.889026    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t" podStartSLOduration=2.240809276 podStartE2EDuration="8.889008421s" totalImagesPullingTime="6.648199145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:54:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:54:19.887149455 +0000 UTC m=+76.918015771" watchObservedRunningTime="2026-09-07 12:54:19.889008421 +0000 UTC m=+76.919874730"
Sep 07 12:54:19.933954 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.933839    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.234112354 podStartE2EDuration="9.93381958s" totalImagesPullingTime="6.673305661s" totalInitContainerRuntime="26.401565ms" isStatefulPod=false podCreationTimestamp="2026-09-07 12:54:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:54:19.931579058 +0000 UTC m=+76.962445380" watchObservedRunningTime="2026-09-07 12:54:19.93381958 +0000 UTC m=+76.964685892"
Sep 07 12:54:19.934759 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.934714    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-5sp6l" podStartSLOduration=4.542991834 podStartE2EDuration="6.934702101s" totalImagesPullingTime="2.391710267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:54:19.904015162 +0000 UTC m=+76.934881475" watchObservedRunningTime="2026-09-07 12:54:19.934702101 +0000 UTC m=+76.965568412"
Sep 07 12:54:19.953949 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:19.953887    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl" podStartSLOduration=4.548631641 podStartE2EDuration="6.953866759s" totalImagesPullingTime="2.405235118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:54:19.95235982 +0000 UTC m=+76.983226131" watchObservedRunningTime="2026-09-07 12:54:19.953866759 +0000 UTC m=+76.984733069"
Sep 07 12:54:21.885073 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:21.885036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ece4bc9d-7296-429a-9029-91b9f6c86689","Type":"ContainerStarted","Data":"26c227fb8ff7a9a445a397fd85ae976741c65c5a37400d342154b96f0c4363bb"}
Sep 07 12:54:21.885480 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:21.885165    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ece4bc9d-7296-429a-9029-91b9f6c86689","Type":"ContainerStarted","Data":"f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700"}
Sep 07 12:54:22.892721 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:22.892685    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ece4bc9d-7296-429a-9029-91b9f6c86689","Type":"ContainerStarted","Data":"bd0e41e0004a8dca96c8da19aaba74bb5d5690f69d2583d62b73453494cba4ac"}
Sep 07 12:54:22.893093 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:22.892752    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ece4bc9d-7296-429a-9029-91b9f6c86689","Type":"ContainerStarted","Data":"af5be1147af6105f116650d93b4696717900d27de2988019ad0d772e33f420de"}
Sep 07 12:54:22.893093 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:22.892782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ece4bc9d-7296-429a-9029-91b9f6c86689","Type":"ContainerStarted","Data":"83ff2a01f8041854c56bbb3e08bb5959e0be0f9120efec0d06879b5d4710afe2"}
Sep 07 12:54:22.893093 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:22.892808    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ece4bc9d-7296-429a-9029-91b9f6c86689","Type":"ContainerStarted","Data":"6fea467fb165d522343edeff359da56115e0349d48f9ea45c2665d9e0c10e6d2"}
Sep 07 12:54:22.929807 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:22.929748    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.055466446 podStartE2EDuration="7.929730637s" totalImagesPullingTime="3.853829709s" totalInitContainerRuntime="20.434482ms" isStatefulPod=false podCreationTimestamp="2026-09-07 12:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:54:22.925916125 +0000 UTC m=+79.956782436" watchObservedRunningTime="2026-09-07 12:54:22.929730637 +0000 UTC m=+79.960596943"
Sep 07 12:54:25.654227 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:25.654096    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:54:25.882928 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:25.882897    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5874f5666d-jdf7t"
Sep 07 12:54:33.814828 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:33.814786    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:33.815298 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:33.814863    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:47.854309 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:47.854272    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5tbdn"
Sep 07 12:54:53.820975 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:53.820940    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:54:53.824785 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:54:53.824762    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-567854bfc5-m9hbl"
Sep 07 12:55:15.653921 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:15.653877    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:15.673460 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:15.673424    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:16.124836 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:16.124803    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:29.518595 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:29.518550    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 12:55:29.519170 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:29.518828    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="alertmanager" containerID="cri-o://8e131ba9eceb43ab0a61b4e3f331dab148e5577360d90b3d7f37e2f71f9c602d" gracePeriod=120
Sep 07 12:55:29.519170 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:29.518905    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="kube-rbac-proxy-metric" containerID="cri-o://31b9f0cc0ddb554d4bb98c177db6578c2769d6407aa01d05d4d846ba5acb0d9b" gracePeriod=120
Sep 07 12:55:29.519170 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:29.518953    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="config-reloader" containerID="cri-o://3d13232ff07457f0c63ef5b3261de83022349ffeac3c167d2da2a1a611f5b6a3" gracePeriod=120
Sep 07 12:55:29.519170 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:29.518982    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="prom-label-proxy" containerID="cri-o://6d135997dfebff46b66d09746ae2005ce52f7a49f42f603ef4150906e7457e0d" gracePeriod=120
Sep 07 12:55:29.519170 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:29.518920    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="kube-rbac-proxy-web" containerID="cri-o://48a333867d35821d21fa0d742affd25b167a0caedc184ec1bf172ec90f22f52b" gracePeriod=120
Sep 07 12:55:29.519170 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:29.518975    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="kube-rbac-proxy" containerID="cri-o://9282c5ddd6b70fade474fe506b38e94ed1777d6ef3338ca2a4aeb623dbadde4a" gracePeriod=120
Sep 07 12:55:30.169463 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.169430    2565 generic.go:356] "Generic (PLEG): container finished" podID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerID="6d135997dfebff46b66d09746ae2005ce52f7a49f42f603ef4150906e7457e0d" exitCode=0
Sep 07 12:55:30.169463 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.169459    2565 generic.go:356] "Generic (PLEG): container finished" podID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerID="31b9f0cc0ddb554d4bb98c177db6578c2769d6407aa01d05d4d846ba5acb0d9b" exitCode=0
Sep 07 12:55:30.169463 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.169465    2565 generic.go:356] "Generic (PLEG): container finished" podID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerID="9282c5ddd6b70fade474fe506b38e94ed1777d6ef3338ca2a4aeb623dbadde4a" exitCode=0
Sep 07 12:55:30.169463 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.169472    2565 generic.go:356] "Generic (PLEG): container finished" podID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerID="3d13232ff07457f0c63ef5b3261de83022349ffeac3c167d2da2a1a611f5b6a3" exitCode=0
Sep 07 12:55:30.169463 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.169479    2565 generic.go:356] "Generic (PLEG): container finished" podID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerID="8e131ba9eceb43ab0a61b4e3f331dab148e5577360d90b3d7f37e2f71f9c602d" exitCode=0
Sep 07 12:55:30.169763 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.169499    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16","Type":"ContainerDied","Data":"6d135997dfebff46b66d09746ae2005ce52f7a49f42f603ef4150906e7457e0d"}
Sep 07 12:55:30.169763 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.169588    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16","Type":"ContainerDied","Data":"31b9f0cc0ddb554d4bb98c177db6578c2769d6407aa01d05d4d846ba5acb0d9b"}
Sep 07 12:55:30.169763 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.169614    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16","Type":"ContainerDied","Data":"9282c5ddd6b70fade474fe506b38e94ed1777d6ef3338ca2a4aeb623dbadde4a"}
Sep 07 12:55:30.169763 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.169638    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16","Type":"ContainerDied","Data":"3d13232ff07457f0c63ef5b3261de83022349ffeac3c167d2da2a1a611f5b6a3"}
Sep 07 12:55:30.169763 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.169662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16","Type":"ContainerDied","Data":"8e131ba9eceb43ab0a61b4e3f331dab148e5577360d90b3d7f37e2f71f9c602d"}
Sep 07 12:55:30.769605 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.769582    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:30.954046 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.953955    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-config-volume\") pod \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") "
Sep 07 12:55:30.954046 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.953993    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy-web\") pod \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") "
Sep 07 12:55:30.954046 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.954016    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy\") pod \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") "
Sep 07 12:55:30.954046 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.954040    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-main-tls\") pod \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") "
Sep 07 12:55:30.954444 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.954087    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy-metric\") pod \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") "
Sep 07 12:55:30.954444 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.954149    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-alertmanager-trusted-ca-bundle\") pod \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") "
Sep 07 12:55:30.954444 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.954177    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-alertmanager-main-db\") pod \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") "
Sep 07 12:55:30.954601 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.954536    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-config-out\") pod \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") "
Sep 07 12:55:30.954601 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.954550    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" (UID: "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 12:55:30.954715 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.954595    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-tls-assets\") pod \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") "
Sep 07 12:55:30.954715 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.954662    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-cluster-tls-config\") pod \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") "
Sep 07 12:55:30.954715 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.954701    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-web-config\") pod \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") "
Sep 07 12:55:30.954875 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.954744    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-metrics-client-ca\") pod \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") "
Sep 07 12:55:30.954875 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.954746    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" (UID: "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 12:55:30.954875 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.954779    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c2p4\" (UniqueName: \"kubernetes.io/projected/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-kube-api-access-6c2p4\") pod \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\" (UID: \"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16\") "
Sep 07 12:55:30.955071 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.955050    2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:30.955159 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.955076    2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-alertmanager-main-db\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:30.955576 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.955524    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" (UID: "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 12:55:30.957730 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.957682    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-config-volume" (OuterVolumeSpecName: "config-volume") pod "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" (UID: "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 12:55:30.957862 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.957725    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" (UID: "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 12:55:30.958281 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.958226    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-config-out" (OuterVolumeSpecName: "config-out") pod "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" (UID: "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 12:55:30.958281 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.958250    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" (UID: "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 12:55:30.958281 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.958264    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-kube-api-access-6c2p4" (OuterVolumeSpecName: "kube-api-access-6c2p4") pod "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" (UID: "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16"). InnerVolumeSpecName "kube-api-access-6c2p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 12:55:30.958512 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.958319    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" (UID: "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 12:55:30.958512 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.958409    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" (UID: "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 12:55:30.958512 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.958492    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" (UID: "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 12:55:30.963066 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.963040    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" (UID: "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 12:55:30.969853 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:30.969830    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-web-config" (OuterVolumeSpecName: "web-config") pod "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" (UID: "25fea9ce-cdf0-4b1c-86a9-8aa64075ea16"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 12:55:31.056170 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.056126    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c2p4\" (UniqueName: \"kubernetes.io/projected/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-kube-api-access-6c2p4\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:31.056170 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.056162    2565 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:31.056170 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.056175    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:31.056436 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.056185    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:31.056436 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.056193    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-main-tls\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:31.056436 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.056202    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:31.056436 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.056211    2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-config-out\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:31.056436 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.056219    2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-tls-assets\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:31.056436 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.056226    2565 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-cluster-tls-config\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:31.056436 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.056235    2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-web-config\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:31.056436 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.056243    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16-metrics-client-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:31.177505 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.177462    2565 generic.go:356] "Generic (PLEG): container finished" podID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerID="48a333867d35821d21fa0d742affd25b167a0caedc184ec1bf172ec90f22f52b" exitCode=0
Sep 07 12:55:31.177688 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.177510    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16","Type":"ContainerDied","Data":"48a333867d35821d21fa0d742affd25b167a0caedc184ec1bf172ec90f22f52b"}
Sep 07 12:55:31.177688 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.177537    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.177688 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.177588    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25fea9ce-cdf0-4b1c-86a9-8aa64075ea16","Type":"ContainerDied","Data":"28490d8ec337c0c52c1fc95871a4e43fd8100b36967e2ad30cb81e077474deb2"}
Sep 07 12:55:31.177688 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.177647    2565 scope.go:117] "RemoveContainer" containerID="6d135997dfebff46b66d09746ae2005ce52f7a49f42f603ef4150906e7457e0d"
Sep 07 12:55:31.185951 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.185932    2565 scope.go:117] "RemoveContainer" containerID="31b9f0cc0ddb554d4bb98c177db6578c2769d6407aa01d05d4d846ba5acb0d9b"
Sep 07 12:55:31.193032 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.193015    2565 scope.go:117] "RemoveContainer" containerID="9282c5ddd6b70fade474fe506b38e94ed1777d6ef3338ca2a4aeb623dbadde4a"
Sep 07 12:55:31.199700 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.199682    2565 scope.go:117] "RemoveContainer" containerID="48a333867d35821d21fa0d742affd25b167a0caedc184ec1bf172ec90f22f52b"
Sep 07 12:55:31.207088 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.207065    2565 scope.go:117] "RemoveContainer" containerID="3d13232ff07457f0c63ef5b3261de83022349ffeac3c167d2da2a1a611f5b6a3"
Sep 07 12:55:31.207637 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.207617    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 12:55:31.214860 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.214838    2565 scope.go:117] "RemoveContainer" containerID="8e131ba9eceb43ab0a61b4e3f331dab148e5577360d90b3d7f37e2f71f9c602d"
Sep 07 12:55:31.216096 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.216074    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 12:55:31.222027 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.222011    2565 scope.go:117] "RemoveContainer" containerID="9f778aeaf6351c863504ca3a94effe99f0bd8b1e1ee26ba9c6aeff8c77281ce5"
Sep 07 12:55:31.228793 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.228779    2565 scope.go:117] "RemoveContainer" containerID="6d135997dfebff46b66d09746ae2005ce52f7a49f42f603ef4150906e7457e0d"
Sep 07 12:55:31.229073 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:31.229046    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d135997dfebff46b66d09746ae2005ce52f7a49f42f603ef4150906e7457e0d\": container with ID starting with 6d135997dfebff46b66d09746ae2005ce52f7a49f42f603ef4150906e7457e0d not found: ID does not exist" containerID="6d135997dfebff46b66d09746ae2005ce52f7a49f42f603ef4150906e7457e0d"
Sep 07 12:55:31.229158 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.229082    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d135997dfebff46b66d09746ae2005ce52f7a49f42f603ef4150906e7457e0d"} err="failed to get container status \"6d135997dfebff46b66d09746ae2005ce52f7a49f42f603ef4150906e7457e0d\": rpc error: code = NotFound desc = could not find container \"6d135997dfebff46b66d09746ae2005ce52f7a49f42f603ef4150906e7457e0d\": container with ID starting with 6d135997dfebff46b66d09746ae2005ce52f7a49f42f603ef4150906e7457e0d not found: ID does not exist"
Sep 07 12:55:31.229158 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.229115    2565 scope.go:117] "RemoveContainer" containerID="31b9f0cc0ddb554d4bb98c177db6578c2769d6407aa01d05d4d846ba5acb0d9b"
Sep 07 12:55:31.229372 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:31.229354    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b9f0cc0ddb554d4bb98c177db6578c2769d6407aa01d05d4d846ba5acb0d9b\": container with ID starting with 31b9f0cc0ddb554d4bb98c177db6578c2769d6407aa01d05d4d846ba5acb0d9b not found: ID does not exist" containerID="31b9f0cc0ddb554d4bb98c177db6578c2769d6407aa01d05d4d846ba5acb0d9b"
Sep 07 12:55:31.229418 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.229379    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b9f0cc0ddb554d4bb98c177db6578c2769d6407aa01d05d4d846ba5acb0d9b"} err="failed to get container status \"31b9f0cc0ddb554d4bb98c177db6578c2769d6407aa01d05d4d846ba5acb0d9b\": rpc error: code = NotFound desc = could not find container \"31b9f0cc0ddb554d4bb98c177db6578c2769d6407aa01d05d4d846ba5acb0d9b\": container with ID starting with 31b9f0cc0ddb554d4bb98c177db6578c2769d6407aa01d05d4d846ba5acb0d9b not found: ID does not exist"
Sep 07 12:55:31.229418 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.229395    2565 scope.go:117] "RemoveContainer" containerID="9282c5ddd6b70fade474fe506b38e94ed1777d6ef3338ca2a4aeb623dbadde4a"
Sep 07 12:55:31.229603 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:31.229588    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9282c5ddd6b70fade474fe506b38e94ed1777d6ef3338ca2a4aeb623dbadde4a\": container with ID starting with 9282c5ddd6b70fade474fe506b38e94ed1777d6ef3338ca2a4aeb623dbadde4a not found: ID does not exist" containerID="9282c5ddd6b70fade474fe506b38e94ed1777d6ef3338ca2a4aeb623dbadde4a"
Sep 07 12:55:31.229644 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.229606    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9282c5ddd6b70fade474fe506b38e94ed1777d6ef3338ca2a4aeb623dbadde4a"} err="failed to get container status \"9282c5ddd6b70fade474fe506b38e94ed1777d6ef3338ca2a4aeb623dbadde4a\": rpc error: code = NotFound desc = could not find container \"9282c5ddd6b70fade474fe506b38e94ed1777d6ef3338ca2a4aeb623dbadde4a\": container with ID starting with 9282c5ddd6b70fade474fe506b38e94ed1777d6ef3338ca2a4aeb623dbadde4a not found: ID does not exist"
Sep 07 12:55:31.229644 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.229618    2565 scope.go:117] "RemoveContainer" containerID="48a333867d35821d21fa0d742affd25b167a0caedc184ec1bf172ec90f22f52b"
Sep 07 12:55:31.229823 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:31.229806    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a333867d35821d21fa0d742affd25b167a0caedc184ec1bf172ec90f22f52b\": container with ID starting with 48a333867d35821d21fa0d742affd25b167a0caedc184ec1bf172ec90f22f52b not found: ID does not exist" containerID="48a333867d35821d21fa0d742affd25b167a0caedc184ec1bf172ec90f22f52b"
Sep 07 12:55:31.229862 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.229830    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a333867d35821d21fa0d742affd25b167a0caedc184ec1bf172ec90f22f52b"} err="failed to get container status \"48a333867d35821d21fa0d742affd25b167a0caedc184ec1bf172ec90f22f52b\": rpc error: code = NotFound desc = could not find container \"48a333867d35821d21fa0d742affd25b167a0caedc184ec1bf172ec90f22f52b\": container with ID starting with 48a333867d35821d21fa0d742affd25b167a0caedc184ec1bf172ec90f22f52b not found: ID does not exist"
Sep 07 12:55:31.229862 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.229843    2565 scope.go:117] "RemoveContainer" containerID="3d13232ff07457f0c63ef5b3261de83022349ffeac3c167d2da2a1a611f5b6a3"
Sep 07 12:55:31.230088 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:31.230072    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d13232ff07457f0c63ef5b3261de83022349ffeac3c167d2da2a1a611f5b6a3\": container with ID starting with 3d13232ff07457f0c63ef5b3261de83022349ffeac3c167d2da2a1a611f5b6a3 not found: ID does not exist" containerID="3d13232ff07457f0c63ef5b3261de83022349ffeac3c167d2da2a1a611f5b6a3"
Sep 07 12:55:31.230137 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.230089    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d13232ff07457f0c63ef5b3261de83022349ffeac3c167d2da2a1a611f5b6a3"} err="failed to get container status \"3d13232ff07457f0c63ef5b3261de83022349ffeac3c167d2da2a1a611f5b6a3\": rpc error: code = NotFound desc = could not find container \"3d13232ff07457f0c63ef5b3261de83022349ffeac3c167d2da2a1a611f5b6a3\": container with ID starting with 3d13232ff07457f0c63ef5b3261de83022349ffeac3c167d2da2a1a611f5b6a3 not found: ID does not exist"
Sep 07 12:55:31.230137 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.230101    2565 scope.go:117] "RemoveContainer" containerID="8e131ba9eceb43ab0a61b4e3f331dab148e5577360d90b3d7f37e2f71f9c602d"
Sep 07 12:55:31.230298 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:31.230282    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e131ba9eceb43ab0a61b4e3f331dab148e5577360d90b3d7f37e2f71f9c602d\": container with ID starting with 8e131ba9eceb43ab0a61b4e3f331dab148e5577360d90b3d7f37e2f71f9c602d not found: ID does not exist" containerID="8e131ba9eceb43ab0a61b4e3f331dab148e5577360d90b3d7f37e2f71f9c602d"
Sep 07 12:55:31.230364 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.230301    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e131ba9eceb43ab0a61b4e3f331dab148e5577360d90b3d7f37e2f71f9c602d"} err="failed to get container status \"8e131ba9eceb43ab0a61b4e3f331dab148e5577360d90b3d7f37e2f71f9c602d\": rpc error: code = NotFound desc = could not find container \"8e131ba9eceb43ab0a61b4e3f331dab148e5577360d90b3d7f37e2f71f9c602d\": container with ID starting with 8e131ba9eceb43ab0a61b4e3f331dab148e5577360d90b3d7f37e2f71f9c602d not found: ID does not exist"
Sep 07 12:55:31.230364 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.230313    2565 scope.go:117] "RemoveContainer" containerID="9f778aeaf6351c863504ca3a94effe99f0bd8b1e1ee26ba9c6aeff8c77281ce5"
Sep 07 12:55:31.230580 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:31.230564    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f778aeaf6351c863504ca3a94effe99f0bd8b1e1ee26ba9c6aeff8c77281ce5\": container with ID starting with 9f778aeaf6351c863504ca3a94effe99f0bd8b1e1ee26ba9c6aeff8c77281ce5 not found: ID does not exist" containerID="9f778aeaf6351c863504ca3a94effe99f0bd8b1e1ee26ba9c6aeff8c77281ce5"
Sep 07 12:55:31.230631 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.230584    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f778aeaf6351c863504ca3a94effe99f0bd8b1e1ee26ba9c6aeff8c77281ce5"} err="failed to get container status \"9f778aeaf6351c863504ca3a94effe99f0bd8b1e1ee26ba9c6aeff8c77281ce5\": rpc error: code = NotFound desc = could not find container \"9f778aeaf6351c863504ca3a94effe99f0bd8b1e1ee26ba9c6aeff8c77281ce5\": container with ID starting with 9f778aeaf6351c863504ca3a94effe99f0bd8b1e1ee26ba9c6aeff8c77281ce5 not found: ID does not exist"
Sep 07 12:55:31.246479 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.246454    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 12:55:31.246803 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.246785    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="alertmanager"
Sep 07 12:55:31.246892 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.246806    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="alertmanager"
Sep 07 12:55:31.246892 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.246830    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="kube-rbac-proxy-web"
Sep 07 12:55:31.246892 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.246839    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="kube-rbac-proxy-web"
Sep 07 12:55:31.246892 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.246853    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="config-reloader"
Sep 07 12:55:31.246892 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.246862    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="config-reloader"
Sep 07 12:55:31.246892 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.246872    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="kube-rbac-proxy-metric"
Sep 07 12:55:31.246892 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.246881    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="kube-rbac-proxy-metric"
Sep 07 12:55:31.246892 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.246892    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="kube-rbac-proxy"
Sep 07 12:55:31.247254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.246900    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="kube-rbac-proxy"
Sep 07 12:55:31.247254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.246915    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="init-config-reloader"
Sep 07 12:55:31.247254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.246924    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="init-config-reloader"
Sep 07 12:55:31.247254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.246937    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="prom-label-proxy"
Sep 07 12:55:31.247254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.246945    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="prom-label-proxy"
Sep 07 12:55:31.247254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.247023    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="prom-label-proxy"
Sep 07 12:55:31.247254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.247039    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="alertmanager"
Sep 07 12:55:31.247254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.247047    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="kube-rbac-proxy-web"
Sep 07 12:55:31.247254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.247056    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="kube-rbac-proxy"
Sep 07 12:55:31.247254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.247065    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="kube-rbac-proxy-metric"
Sep 07 12:55:31.247254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.247078    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" containerName="config-reloader"
Sep 07 12:55:31.252169 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.252151    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.254728 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.254709    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 07 12:55:31.254842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.254796    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 07 12:55:31.254842 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.254813    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 07 12:55:31.254958 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.254815    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 07 12:55:31.255123 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.255102    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 07 12:55:31.255234 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.255102    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 07 12:55:31.255304 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.255104    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4kk8g\""
Sep 07 12:55:31.255414 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.255112    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 07 12:55:31.255414 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.255143    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 07 12:55:31.259665 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.259647    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 07 12:55:31.266178 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.265389    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 12:55:31.357782 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.357747    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f785a771-6a28-441e-a35a-bb427caaf8c7-config-out\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.357782 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.357779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f785a771-6a28-441e-a35a-bb427caaf8c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.357982 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.357798    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.357982 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.357826    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.357982 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.357893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f785a771-6a28-441e-a35a-bb427caaf8c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.357982 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.357953    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-web-config\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.358102 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.357992    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fms76\" (UniqueName: \"kubernetes.io/projected/f785a771-6a28-441e-a35a-bb427caaf8c7-kube-api-access-fms76\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.358102 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.358016    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f785a771-6a28-441e-a35a-bb427caaf8c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.358102 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.358038    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.358102 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.358066    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.358102 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.358095    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.358243 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.358122    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f785a771-6a28-441e-a35a-bb427caaf8c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.358243 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.358144    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.459172 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.459086    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fms76\" (UniqueName: \"kubernetes.io/projected/f785a771-6a28-441e-a35a-bb427caaf8c7-kube-api-access-fms76\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.459172 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.459129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f785a771-6a28-441e-a35a-bb427caaf8c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.459172 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.459154    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.459458 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.459266    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.459458 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.459317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.459458 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.459393    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f785a771-6a28-441e-a35a-bb427caaf8c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.459458 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.459412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.459458 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.459437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f785a771-6a28-441e-a35a-bb427caaf8c7-config-out\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.459458 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.459460    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f785a771-6a28-441e-a35a-bb427caaf8c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.459733 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.459490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.459733 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.459520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.459733 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.459545    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f785a771-6a28-441e-a35a-bb427caaf8c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.459733 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.459563    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f785a771-6a28-441e-a35a-bb427caaf8c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.459733 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.459584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-web-config\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.460671 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.460644    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f785a771-6a28-441e-a35a-bb427caaf8c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.462371 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.462347    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f785a771-6a28-441e-a35a-bb427caaf8c7-config-out\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.462585 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.462523    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.462585 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.462523    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.462803 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.462673    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-web-config\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.462803 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.462749    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.462926 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.462868    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f785a771-6a28-441e-a35a-bb427caaf8c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.463116 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.463092    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.463199 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.463168    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.463311 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.463289    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f785a771-6a28-441e-a35a-bb427caaf8c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.464446 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.464427    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f785a771-6a28-441e-a35a-bb427caaf8c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.476551 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.476528    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fms76\" (UniqueName: \"kubernetes.io/projected/f785a771-6a28-441e-a35a-bb427caaf8c7-kube-api-access-fms76\") pod \"alertmanager-main-0\" (UID: \"f785a771-6a28-441e-a35a-bb427caaf8c7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.487117 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.487083    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25fea9ce-cdf0-4b1c-86a9-8aa64075ea16" path="/var/lib/kubelet/pods/25fea9ce-cdf0-4b1c-86a9-8aa64075ea16/volumes"
Sep 07 12:55:31.561571 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.561532    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 12:55:31.858385 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:31.857916    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 12:55:31.860817 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:55:31.860785    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf785a771_6a28_441e_a35a_bb427caaf8c7.slice/crio-905ece6198b551fef43f6f3768091b13ba1ab0eb755c2203492905fa131667e8 WatchSource:0}: Error finding container 905ece6198b551fef43f6f3768091b13ba1ab0eb755c2203492905fa131667e8: Status 404 returned error can't find the container with id 905ece6198b551fef43f6f3768091b13ba1ab0eb755c2203492905fa131667e8
Sep 07 12:55:32.188419 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:32.188311    2565 generic.go:356] "Generic (PLEG): container finished" podID="f785a771-6a28-441e-a35a-bb427caaf8c7" containerID="1ca50f04e873d7b352743042d8c20e21809b948d7646a4e0b94cf00765106205" exitCode=0
Sep 07 12:55:32.188419 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:32.188364    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f785a771-6a28-441e-a35a-bb427caaf8c7","Type":"ContainerDied","Data":"1ca50f04e873d7b352743042d8c20e21809b948d7646a4e0b94cf00765106205"}
Sep 07 12:55:32.188608 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:32.188463    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f785a771-6a28-441e-a35a-bb427caaf8c7","Type":"ContainerStarted","Data":"905ece6198b551fef43f6f3768091b13ba1ab0eb755c2203492905fa131667e8"}
Sep 07 12:55:33.196860 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.196818    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f785a771-6a28-441e-a35a-bb427caaf8c7","Type":"ContainerStarted","Data":"ad09ab561e5fab472b013d919c7d32ef399c6f7c7ee29c8a97782bc7222cded2"}
Sep 07 12:55:33.197289 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.196888    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f785a771-6a28-441e-a35a-bb427caaf8c7","Type":"ContainerStarted","Data":"a7f66f87acbdca02cf4a4096ed62996e5f0a7afcca0864377329f3d8cd393f4c"}
Sep 07 12:55:33.197289 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.196916    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f785a771-6a28-441e-a35a-bb427caaf8c7","Type":"ContainerStarted","Data":"532273e07896805517a86669d0016d52f4812db86ceeeefcced7fa48b468b380"}
Sep 07 12:55:33.197289 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.196954    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f785a771-6a28-441e-a35a-bb427caaf8c7","Type":"ContainerStarted","Data":"618ff937adbffd282ba84535b376b443fa71c1dcb933f802012e4f0a7651d71a"}
Sep 07 12:55:33.197289 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.196984    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f785a771-6a28-441e-a35a-bb427caaf8c7","Type":"ContainerStarted","Data":"6019ccd6fe0cd07155acce3ea44ae851233329e4ec30b1e8b4448932a648a392"}
Sep 07 12:55:33.197289 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.197009    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f785a771-6a28-441e-a35a-bb427caaf8c7","Type":"ContainerStarted","Data":"8d3af100b13d3b1fbe58c3b30d023fd7a9ec865291bcc35ba4c127ad3a23d1d9"}
Sep 07 12:55:33.230480 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.230423    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.212687171 podStartE2EDuration="2.230405177s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.718006ms" isStatefulPod=false podCreationTimestamp="2026-09-07 12:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:55:33.228444162 +0000 UTC m=+150.259310471" watchObservedRunningTime="2026-09-07 12:55:33.230405177 +0000 UTC m=+150.261271468"
Sep 07 12:55:33.634791 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.634754    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"]
Sep 07 12:55:33.638535 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.638512    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.641224 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.641172    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 07 12:55:33.642254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.641682    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 07 12:55:33.642254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.641714    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 07 12:55:33.642254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.641949    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 07 12:55:33.642254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.642118    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 07 12:55:33.642562 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.642378    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4zbgk\""
Sep 07 12:55:33.650574 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.650555    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 07 12:55:33.721885 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.721845    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"]
Sep 07 12:55:33.781080 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.781037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e62da2fb-574e-48fd-b94b-ea0e50807ce0-secret-telemeter-client\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.781080 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.781078    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e62da2fb-574e-48fd-b94b-ea0e50807ce0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.781316 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.781109    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e62da2fb-574e-48fd-b94b-ea0e50807ce0-serving-certs-ca-bundle\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.781316 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.781173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e62da2fb-574e-48fd-b94b-ea0e50807ce0-metrics-client-ca\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.781316 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.781213    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e62da2fb-574e-48fd-b94b-ea0e50807ce0-telemeter-client-tls\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.781316 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.781234    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e62da2fb-574e-48fd-b94b-ea0e50807ce0-federate-client-tls\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.781316 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.781249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e62da2fb-574e-48fd-b94b-ea0e50807ce0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.781316 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.781265    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttnj9\" (UniqueName: \"kubernetes.io/projected/e62da2fb-574e-48fd-b94b-ea0e50807ce0-kube-api-access-ttnj9\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.881881 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.881841    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e62da2fb-574e-48fd-b94b-ea0e50807ce0-serving-certs-ca-bundle\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.882053 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.881888    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e62da2fb-574e-48fd-b94b-ea0e50807ce0-metrics-client-ca\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.882053 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.881914    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e62da2fb-574e-48fd-b94b-ea0e50807ce0-telemeter-client-tls\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.882053 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.882036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e62da2fb-574e-48fd-b94b-ea0e50807ce0-federate-client-tls\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.882201 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.882078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e62da2fb-574e-48fd-b94b-ea0e50807ce0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.882201 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.882103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttnj9\" (UniqueName: \"kubernetes.io/projected/e62da2fb-574e-48fd-b94b-ea0e50807ce0-kube-api-access-ttnj9\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.882296 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.882214    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e62da2fb-574e-48fd-b94b-ea0e50807ce0-secret-telemeter-client\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.882296 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.882242    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e62da2fb-574e-48fd-b94b-ea0e50807ce0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.882777 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.882743    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e62da2fb-574e-48fd-b94b-ea0e50807ce0-serving-certs-ca-bundle\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.883058 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.882743    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e62da2fb-574e-48fd-b94b-ea0e50807ce0-metrics-client-ca\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.883524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.883502    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e62da2fb-574e-48fd-b94b-ea0e50807ce0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.884785 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.884758    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e62da2fb-574e-48fd-b94b-ea0e50807ce0-federate-client-tls\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.884877 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.884832    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e62da2fb-574e-48fd-b94b-ea0e50807ce0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.885099 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.885080    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e62da2fb-574e-48fd-b94b-ea0e50807ce0-secret-telemeter-client\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.885159 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.885137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e62da2fb-574e-48fd-b94b-ea0e50807ce0-telemeter-client-tls\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.895792 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.895773    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttnj9\" (UniqueName: \"kubernetes.io/projected/e62da2fb-574e-48fd-b94b-ea0e50807ce0-kube-api-access-ttnj9\") pod \"telemeter-client-8c5dd6dc4-mlb66\" (UID: \"e62da2fb-574e-48fd-b94b-ea0e50807ce0\") " pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.950944 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.950905    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"
Sep 07 12:55:33.984698 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.984660    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 12:55:33.985042 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.985000    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="prometheus" containerID="cri-o://f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700" gracePeriod=600
Sep 07 12:55:33.985134 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.985022    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="kube-rbac-proxy" containerID="cri-o://af5be1147af6105f116650d93b4696717900d27de2988019ad0d772e33f420de" gracePeriod=600
Sep 07 12:55:33.985134 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.985089    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="config-reloader" containerID="cri-o://26c227fb8ff7a9a445a397fd85ae976741c65c5a37400d342154b96f0c4363bb" gracePeriod=600
Sep 07 12:55:33.985250 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.985052    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="kube-rbac-proxy-web" containerID="cri-o://83ff2a01f8041854c56bbb3e08bb5959e0be0f9120efec0d06879b5d4710afe2" gracePeriod=600
Sep 07 12:55:33.985250 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.985025    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="thanos-sidecar" containerID="cri-o://6fea467fb165d522343edeff359da56115e0349d48f9ea45c2665d9e0c10e6d2" gracePeriod=600
Sep 07 12:55:33.985377 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:33.985203    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="kube-rbac-proxy-thanos" containerID="cri-o://bd0e41e0004a8dca96c8da19aaba74bb5d5690f69d2583d62b73453494cba4ac" gracePeriod=600
Sep 07 12:55:34.206170 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:34.206068    2565 generic.go:356] "Generic (PLEG): container finished" podID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerID="6fea467fb165d522343edeff359da56115e0349d48f9ea45c2665d9e0c10e6d2" exitCode=0
Sep 07 12:55:34.206170 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:34.206100    2565 generic.go:356] "Generic (PLEG): container finished" podID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerID="26c227fb8ff7a9a445a397fd85ae976741c65c5a37400d342154b96f0c4363bb" exitCode=0
Sep 07 12:55:34.207294 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:34.207268    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ece4bc9d-7296-429a-9029-91b9f6c86689","Type":"ContainerDied","Data":"6fea467fb165d522343edeff359da56115e0349d48f9ea45c2665d9e0c10e6d2"}
Sep 07 12:55:34.207427 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:34.207362    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ece4bc9d-7296-429a-9029-91b9f6c86689","Type":"ContainerDied","Data":"26c227fb8ff7a9a445a397fd85ae976741c65c5a37400d342154b96f0c4363bb"}
Sep 07 12:55:34.270248 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:34.270208    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66"]
Sep 07 12:55:34.274498 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:55:34.274465    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62da2fb_574e_48fd_b94b_ea0e50807ce0.slice/crio-9035851d0a8422d2a1b69a274b1b91894e576804ca85ce4e9d801ff1886bb311 WatchSource:0}: Error finding container 9035851d0a8422d2a1b69a274b1b91894e576804ca85ce4e9d801ff1886bb311: Status 404 returned error can't find the container with id 9035851d0a8422d2a1b69a274b1b91894e576804ca85ce4e9d801ff1886bb311
Sep 07 12:55:35.215315 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:35.215222    2565 generic.go:356] "Generic (PLEG): container finished" podID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerID="bd0e41e0004a8dca96c8da19aaba74bb5d5690f69d2583d62b73453494cba4ac" exitCode=0
Sep 07 12:55:35.215315 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:35.215253    2565 generic.go:356] "Generic (PLEG): container finished" podID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerID="af5be1147af6105f116650d93b4696717900d27de2988019ad0d772e33f420de" exitCode=0
Sep 07 12:55:35.215315 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:35.215264    2565 generic.go:356] "Generic (PLEG): container finished" podID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerID="83ff2a01f8041854c56bbb3e08bb5959e0be0f9120efec0d06879b5d4710afe2" exitCode=0
Sep 07 12:55:35.215315 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:35.215291    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ece4bc9d-7296-429a-9029-91b9f6c86689","Type":"ContainerDied","Data":"bd0e41e0004a8dca96c8da19aaba74bb5d5690f69d2583d62b73453494cba4ac"}
Sep 07 12:55:35.215896 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:35.215400    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ece4bc9d-7296-429a-9029-91b9f6c86689","Type":"ContainerDied","Data":"af5be1147af6105f116650d93b4696717900d27de2988019ad0d772e33f420de"}
Sep 07 12:55:35.215896 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:35.215444    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ece4bc9d-7296-429a-9029-91b9f6c86689","Type":"ContainerDied","Data":"83ff2a01f8041854c56bbb3e08bb5959e0be0f9120efec0d06879b5d4710afe2"}
Sep 07 12:55:35.216701 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:35.216678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66" event={"ID":"e62da2fb-574e-48fd-b94b-ea0e50807ce0","Type":"ContainerStarted","Data":"9035851d0a8422d2a1b69a274b1b91894e576804ca85ce4e9d801ff1886bb311"}
Sep 07 12:55:36.109548 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:36.109511    2565 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Sep 07 12:55:36.110455 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:36.110424    2565 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Sep 07 12:55:36.111355 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:36.111315    2565 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Sep 07 12:55:36.111438 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:36.111367    2565 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="prometheus" probeResult="unknown"
Sep 07 12:55:36.222788 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:36.222763    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66" event={"ID":"e62da2fb-574e-48fd-b94b-ea0e50807ce0","Type":"ContainerStarted","Data":"c2d51b5c62ac1728d746566f0cf7ccfffc94527e0b75ae592c00f50e90f7e15c"}
Sep 07 12:55:36.223084 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:36.222826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66" event={"ID":"e62da2fb-574e-48fd-b94b-ea0e50807ce0","Type":"ContainerStarted","Data":"0123f951c1e1e6bc73ab23b6c0c7dd1ced839d60d98d1d4b500e8d09c9b5d5d1"}
Sep 07 12:55:37.228760 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:37.228716    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66" event={"ID":"e62da2fb-574e-48fd-b94b-ea0e50807ce0","Type":"ContainerStarted","Data":"c4d623d338d93b41a5cd64f6583648721f7ef8ae82a9411de061500c51d9acc7"}
Sep 07 12:55:37.252856 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:37.252800    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8c5dd6dc4-mlb66" podStartSLOduration=2.44925098 podStartE2EDuration="4.252777768s" totalImagesPullingTime="1.803526788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:55:37.2507381 +0000 UTC m=+154.281604410" watchObservedRunningTime="2026-09-07 12:55:37.252777768 +0000 UTC m=+154.283644080"
Sep 07 12:55:41.110347 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:41.110282    2565 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Sep 07 12:55:41.111314 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:41.111272    2565 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Sep 07 12:55:41.112270 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:41.112238    2565 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Sep 07 12:55:41.112381 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:41.112283    2565 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="prometheus" probeResult="unknown"
Sep 07 12:55:41.935516 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:41.935490    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.057990 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.057959    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-metrics-client-ca\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058171 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058004    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-serving-certs-ca-bundle\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058171 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058035    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqllb\" (UniqueName: \"kubernetes.io/projected/ece4bc9d-7296-429a-9029-91b9f6c86689-kube-api-access-tqllb\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058171 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058059    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-grpc-tls\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058348 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058245    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-tls\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058348 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058282    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ece4bc9d-7296-429a-9029-91b9f6c86689-tls-assets\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058348 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058337    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-trusted-ca-bundle\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058367    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-thanos-prometheus-http-client-file\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058394    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 12:55:42.058524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058408    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-web-config\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058437    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 12:55:42.058524 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058469    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-metrics-client-certs\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058978 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058530    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-kubelet-serving-ca-bundle\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058978 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058558    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058978 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058604    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-db\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058978 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058630    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-1\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058978 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058669    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ece4bc9d-7296-429a-9029-91b9f6c86689-config-out\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058978 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058709    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-0\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058978 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058739    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.058978 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058849    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 12:55:42.058978 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058900    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 12:55:42.058978 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058952    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-kube-rbac-proxy\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.059780 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.058987    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-2\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.059780 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.059011    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-config\") pod \"ece4bc9d-7296-429a-9029-91b9f6c86689\" (UID: \"ece4bc9d-7296-429a-9029-91b9f6c86689\") "
Sep 07 12:55:42.059780 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.059297    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.059780 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.059320    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-metrics-client-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.059780 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.059468    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.059780 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.059484    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.060213 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.059943    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 12:55:42.060269 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.060251    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 12:55:42.061196 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.061153    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 12:55:42.061367 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.061343    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 12:55:42.062075 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.062040    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 12:55:42.062498 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.062390    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 12:55:42.062498 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.062475    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece4bc9d-7296-429a-9029-91b9f6c86689-kube-api-access-tqllb" (OuterVolumeSpecName: "kube-api-access-tqllb") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "kube-api-access-tqllb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 12:55:42.062808 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.062778    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece4bc9d-7296-429a-9029-91b9f6c86689-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 12:55:42.063122 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.063094    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-config" (OuterVolumeSpecName: "config") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 12:55:42.063809 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.063774    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 12:55:42.063900 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.063806    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece4bc9d-7296-429a-9029-91b9f6c86689-config-out" (OuterVolumeSpecName: "config-out") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 12:55:42.063965 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.063906    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 12:55:42.064061 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.064036    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 12:55:42.064125 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.064102    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 12:55:42.064896 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.064874    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 12:55:42.073973 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.073952    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-web-config" (OuterVolumeSpecName: "web-config") pod "ece4bc9d-7296-429a-9029-91b9f6c86689" (UID: "ece4bc9d-7296-429a-9029-91b9f6c86689"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 12:55:42.160111 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.160078    2565 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-config\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.160111 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.160109    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqllb\" (UniqueName: \"kubernetes.io/projected/ece4bc9d-7296-429a-9029-91b9f6c86689-kube-api-access-tqllb\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.160582 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.160121    2565 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-grpc-tls\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.160582 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.160131    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.160582 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.160139    2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ece4bc9d-7296-429a-9029-91b9f6c86689-tls-assets\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.160582 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.160148    2565 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.160582 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.160157    2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-web-config\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.160582 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.160166    2565 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-metrics-client-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.160582 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.160174    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.160582 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.160183    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-db\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.160582 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.160191    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.160582 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.160198    2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ece4bc9d-7296-429a-9029-91b9f6c86689-config-out\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.160582 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.160206    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.160582 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.160215    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.160582 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.160224    2565 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ece4bc9d-7296-429a-9029-91b9f6c86689-secret-kube-rbac-proxy\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.160582 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.160231    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ece4bc9d-7296-429a-9029-91b9f6c86689-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 12:55:42.255528 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.255491    2565 generic.go:356] "Generic (PLEG): container finished" podID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerID="f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700" exitCode=0
Sep 07 12:55:42.255695 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.255564    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.255695 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.255579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ece4bc9d-7296-429a-9029-91b9f6c86689","Type":"ContainerDied","Data":"f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700"}
Sep 07 12:55:42.255695 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.255675    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ece4bc9d-7296-429a-9029-91b9f6c86689","Type":"ContainerDied","Data":"454f4e37e5c42b14cd76e4dbc253042a9380f8f965121f9cc4535c7fc7c09241"}
Sep 07 12:55:42.255792 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.255707    2565 scope.go:117] "RemoveContainer" containerID="bd0e41e0004a8dca96c8da19aaba74bb5d5690f69d2583d62b73453494cba4ac"
Sep 07 12:55:42.263924 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.263896    2565 scope.go:117] "RemoveContainer" containerID="af5be1147af6105f116650d93b4696717900d27de2988019ad0d772e33f420de"
Sep 07 12:55:42.271085 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.271068    2565 scope.go:117] "RemoveContainer" containerID="83ff2a01f8041854c56bbb3e08bb5959e0be0f9120efec0d06879b5d4710afe2"
Sep 07 12:55:42.278191 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.278173    2565 scope.go:117] "RemoveContainer" containerID="6fea467fb165d522343edeff359da56115e0349d48f9ea45c2665d9e0c10e6d2"
Sep 07 12:55:42.285695 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.285673    2565 scope.go:117] "RemoveContainer" containerID="26c227fb8ff7a9a445a397fd85ae976741c65c5a37400d342154b96f0c4363bb"
Sep 07 12:55:42.287582 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.287562    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 12:55:42.292841 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.292820    2565 scope.go:117] "RemoveContainer" containerID="f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700"
Sep 07 12:55:42.299922 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.299905    2565 scope.go:117] "RemoveContainer" containerID="a331e53fc20132115118c5dec516fb946fb0dfebac146cafb062eb744b1372df"
Sep 07 12:55:42.307380 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.307306    2565 scope.go:117] "RemoveContainer" containerID="bd0e41e0004a8dca96c8da19aaba74bb5d5690f69d2583d62b73453494cba4ac"
Sep 07 12:55:42.307637 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:42.307611    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0e41e0004a8dca96c8da19aaba74bb5d5690f69d2583d62b73453494cba4ac\": container with ID starting with bd0e41e0004a8dca96c8da19aaba74bb5d5690f69d2583d62b73453494cba4ac not found: ID does not exist" containerID="bd0e41e0004a8dca96c8da19aaba74bb5d5690f69d2583d62b73453494cba4ac"
Sep 07 12:55:42.307699 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.307648    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0e41e0004a8dca96c8da19aaba74bb5d5690f69d2583d62b73453494cba4ac"} err="failed to get container status \"bd0e41e0004a8dca96c8da19aaba74bb5d5690f69d2583d62b73453494cba4ac\": rpc error: code = NotFound desc = could not find container \"bd0e41e0004a8dca96c8da19aaba74bb5d5690f69d2583d62b73453494cba4ac\": container with ID starting with bd0e41e0004a8dca96c8da19aaba74bb5d5690f69d2583d62b73453494cba4ac not found: ID does not exist"
Sep 07 12:55:42.307699 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.307674    2565 scope.go:117] "RemoveContainer" containerID="af5be1147af6105f116650d93b4696717900d27de2988019ad0d772e33f420de"
Sep 07 12:55:42.307788 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.307735    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 12:55:42.307949 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:42.307929    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5be1147af6105f116650d93b4696717900d27de2988019ad0d772e33f420de\": container with ID starting with af5be1147af6105f116650d93b4696717900d27de2988019ad0d772e33f420de not found: ID does not exist" containerID="af5be1147af6105f116650d93b4696717900d27de2988019ad0d772e33f420de"
Sep 07 12:55:42.308048 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.307958    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5be1147af6105f116650d93b4696717900d27de2988019ad0d772e33f420de"} err="failed to get container status \"af5be1147af6105f116650d93b4696717900d27de2988019ad0d772e33f420de\": rpc error: code = NotFound desc = could not find container \"af5be1147af6105f116650d93b4696717900d27de2988019ad0d772e33f420de\": container with ID starting with af5be1147af6105f116650d93b4696717900d27de2988019ad0d772e33f420de not found: ID does not exist"
Sep 07 12:55:42.308048 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.307983    2565 scope.go:117] "RemoveContainer" containerID="83ff2a01f8041854c56bbb3e08bb5959e0be0f9120efec0d06879b5d4710afe2"
Sep 07 12:55:42.308264 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:42.308246    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ff2a01f8041854c56bbb3e08bb5959e0be0f9120efec0d06879b5d4710afe2\": container with ID starting with 83ff2a01f8041854c56bbb3e08bb5959e0be0f9120efec0d06879b5d4710afe2 not found: ID does not exist" containerID="83ff2a01f8041854c56bbb3e08bb5959e0be0f9120efec0d06879b5d4710afe2"
Sep 07 12:55:42.308308 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.308269    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ff2a01f8041854c56bbb3e08bb5959e0be0f9120efec0d06879b5d4710afe2"} err="failed to get container status \"83ff2a01f8041854c56bbb3e08bb5959e0be0f9120efec0d06879b5d4710afe2\": rpc error: code = NotFound desc = could not find container \"83ff2a01f8041854c56bbb3e08bb5959e0be0f9120efec0d06879b5d4710afe2\": container with ID starting with 83ff2a01f8041854c56bbb3e08bb5959e0be0f9120efec0d06879b5d4710afe2 not found: ID does not exist"
Sep 07 12:55:42.308308 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.308283    2565 scope.go:117] "RemoveContainer" containerID="6fea467fb165d522343edeff359da56115e0349d48f9ea45c2665d9e0c10e6d2"
Sep 07 12:55:42.308555 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:42.308540    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fea467fb165d522343edeff359da56115e0349d48f9ea45c2665d9e0c10e6d2\": container with ID starting with 6fea467fb165d522343edeff359da56115e0349d48f9ea45c2665d9e0c10e6d2 not found: ID does not exist" containerID="6fea467fb165d522343edeff359da56115e0349d48f9ea45c2665d9e0c10e6d2"
Sep 07 12:55:42.308601 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.308559    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fea467fb165d522343edeff359da56115e0349d48f9ea45c2665d9e0c10e6d2"} err="failed to get container status \"6fea467fb165d522343edeff359da56115e0349d48f9ea45c2665d9e0c10e6d2\": rpc error: code = NotFound desc = could not find container \"6fea467fb165d522343edeff359da56115e0349d48f9ea45c2665d9e0c10e6d2\": container with ID starting with 6fea467fb165d522343edeff359da56115e0349d48f9ea45c2665d9e0c10e6d2 not found: ID does not exist"
Sep 07 12:55:42.308601 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.308572    2565 scope.go:117] "RemoveContainer" containerID="26c227fb8ff7a9a445a397fd85ae976741c65c5a37400d342154b96f0c4363bb"
Sep 07 12:55:42.308809 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:42.308791    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c227fb8ff7a9a445a397fd85ae976741c65c5a37400d342154b96f0c4363bb\": container with ID starting with 26c227fb8ff7a9a445a397fd85ae976741c65c5a37400d342154b96f0c4363bb not found: ID does not exist" containerID="26c227fb8ff7a9a445a397fd85ae976741c65c5a37400d342154b96f0c4363bb"
Sep 07 12:55:42.308852 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.308814    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c227fb8ff7a9a445a397fd85ae976741c65c5a37400d342154b96f0c4363bb"} err="failed to get container status \"26c227fb8ff7a9a445a397fd85ae976741c65c5a37400d342154b96f0c4363bb\": rpc error: code = NotFound desc = could not find container \"26c227fb8ff7a9a445a397fd85ae976741c65c5a37400d342154b96f0c4363bb\": container with ID starting with 26c227fb8ff7a9a445a397fd85ae976741c65c5a37400d342154b96f0c4363bb not found: ID does not exist"
Sep 07 12:55:42.308852 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.308829    2565 scope.go:117] "RemoveContainer" containerID="f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700"
Sep 07 12:55:42.309049 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:42.309033    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700\": container with ID starting with f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700 not found: ID does not exist" containerID="f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700"
Sep 07 12:55:42.309084 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.309053    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700"} err="failed to get container status \"f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700\": rpc error: code = NotFound desc = could not find container \"f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700\": container with ID starting with f08a2f9873ad4803515d7daf78391bf239f9950001e74e08f0476431145ca700 not found: ID does not exist"
Sep 07 12:55:42.309084 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.309065    2565 scope.go:117] "RemoveContainer" containerID="a331e53fc20132115118c5dec516fb946fb0dfebac146cafb062eb744b1372df"
Sep 07 12:55:42.309302 ip-10-0-141-58 kubenswrapper[2565]: E0907 12:55:42.309284    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a331e53fc20132115118c5dec516fb946fb0dfebac146cafb062eb744b1372df\": container with ID starting with a331e53fc20132115118c5dec516fb946fb0dfebac146cafb062eb744b1372df not found: ID does not exist" containerID="a331e53fc20132115118c5dec516fb946fb0dfebac146cafb062eb744b1372df"
Sep 07 12:55:42.309364 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.309307    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a331e53fc20132115118c5dec516fb946fb0dfebac146cafb062eb744b1372df"} err="failed to get container status \"a331e53fc20132115118c5dec516fb946fb0dfebac146cafb062eb744b1372df\": rpc error: code = NotFound desc = could not find container \"a331e53fc20132115118c5dec516fb946fb0dfebac146cafb062eb744b1372df\": container with ID starting with a331e53fc20132115118c5dec516fb946fb0dfebac146cafb062eb744b1372df not found: ID does not exist"
Sep 07 12:55:42.369957 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.369920    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 12:55:42.370343 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370304    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="prometheus"
Sep 07 12:55:42.370343 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370340    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="prometheus"
Sep 07 12:55:42.370507 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370355    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="thanos-sidecar"
Sep 07 12:55:42.370507 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370363    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="thanos-sidecar"
Sep 07 12:55:42.370507 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370372    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="kube-rbac-proxy-web"
Sep 07 12:55:42.370507 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370389    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="kube-rbac-proxy-web"
Sep 07 12:55:42.370507 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370416    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="kube-rbac-proxy-thanos"
Sep 07 12:55:42.370507 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370424    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="kube-rbac-proxy-thanos"
Sep 07 12:55:42.370507 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370435    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="init-config-reloader"
Sep 07 12:55:42.370507 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370442    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="init-config-reloader"
Sep 07 12:55:42.370507 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370459    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="config-reloader"
Sep 07 12:55:42.370507 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370467    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="config-reloader"
Sep 07 12:55:42.370507 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370477    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="kube-rbac-proxy"
Sep 07 12:55:42.370507 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370485    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="kube-rbac-proxy"
Sep 07 12:55:42.371129 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370572    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="kube-rbac-proxy-web"
Sep 07 12:55:42.371129 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370585    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="kube-rbac-proxy"
Sep 07 12:55:42.371129 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370596    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="prometheus"
Sep 07 12:55:42.371129 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370605    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="config-reloader"
Sep 07 12:55:42.371129 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370614    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="kube-rbac-proxy-thanos"
Sep 07 12:55:42.371129 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.370623    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" containerName="thanos-sidecar"
Sep 07 12:55:42.376420 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.376400    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.383799 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.383775    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 07 12:55:42.383906 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.383871    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 07 12:55:42.384249 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.384226    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 07 12:55:42.384249 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.384246    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 07 12:55:42.384439 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.384290    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 07 12:55:42.384439 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.384235    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d1lvn1p2c7847\""
Sep 07 12:55:42.384591 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.384565    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 07 12:55:42.384677 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.384591    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 07 12:55:42.384677 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.384606    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 07 12:55:42.384677 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.384592    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 07 12:55:42.385208 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.385185    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 07 12:55:42.385312 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.385250    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 07 12:55:42.385312 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.385300    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 07 12:55:42.385455 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.385410    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xdtxc\""
Sep 07 12:55:42.396515 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.396493    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 07 12:55:42.398120 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.398097    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 12:55:42.399204 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.399183    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 07 12:55:42.463099 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463067    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463099 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463102    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/539502f5-568f-4143-a9b5-e2a926f85a08-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463297 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463122    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-config\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463297 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463146    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnvgp\" (UniqueName: \"kubernetes.io/projected/539502f5-568f-4143-a9b5-e2a926f85a08-kube-api-access-bnvgp\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463297 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463171    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463297 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463191    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463297 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463208    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463297 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463237    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463297 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463274    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/539502f5-568f-4143-a9b5-e2a926f85a08-config-out\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463297 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463291    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463574 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463317    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463574 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463363    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463574 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463382    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463574 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463574 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463574 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463574 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463453    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463574 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-web-config\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463574 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463496    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.463574 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.463513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/539502f5-568f-4143-a9b5-e2a926f85a08-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.564066 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.563964    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/539502f5-568f-4143-a9b5-e2a926f85a08-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.564066 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-config\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.564066 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvgp\" (UniqueName: \"kubernetes.io/projected/539502f5-568f-4143-a9b5-e2a926f85a08-kube-api-access-bnvgp\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.564066 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564061    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.564431 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564091    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.564431 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564113    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.564431 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.564431 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/539502f5-568f-4143-a9b5-e2a926f85a08-config-out\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.564431 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.564431 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564225    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.564431 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564247    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.564431 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.564431 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.564431 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564355    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.564431 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.564431 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564421    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.565006 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564449    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-web-config\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.565006 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.565006 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/539502f5-568f-4143-a9b5-e2a926f85a08-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.565006 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.564577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.565218 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.565199    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.565766 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.565737    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.566411 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.566387    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.567554 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.567528    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.567554 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.567528    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.567764 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.567741    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.568457 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.567962    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.568457 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.568285    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.568457 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.568421    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.568457 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.567532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/539502f5-568f-4143-a9b5-e2a926f85a08-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.568732 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.568501    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.569129 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.569054    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/539502f5-568f-4143-a9b5-e2a926f85a08-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.569222 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.569197    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-config\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.569286 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.569201    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.569286 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.569201    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.569820 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.569800    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-web-config\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.569911 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.569877    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/539502f5-568f-4143-a9b5-e2a926f85a08-config-out\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.570423 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.570400    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/539502f5-568f-4143-a9b5-e2a926f85a08-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.571054 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.571037    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/539502f5-568f-4143-a9b5-e2a926f85a08-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.577528 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.577510    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnvgp\" (UniqueName: \"kubernetes.io/projected/539502f5-568f-4143-a9b5-e2a926f85a08-kube-api-access-bnvgp\") pod \"prometheus-k8s-0\" (UID: \"539502f5-568f-4143-a9b5-e2a926f85a08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.687770 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.687734    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:55:42.999736 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:42.999704    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 12:55:43.002074 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:55:43.002046    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539502f5_568f_4143_a9b5_e2a926f85a08.slice/crio-52dbd6c42bba2a2b15018f616c2a0b33700ee7d9962e46c0cbef10cdd95d6063 WatchSource:0}: Error finding container 52dbd6c42bba2a2b15018f616c2a0b33700ee7d9962e46c0cbef10cdd95d6063: Status 404 returned error can't find the container with id 52dbd6c42bba2a2b15018f616c2a0b33700ee7d9962e46c0cbef10cdd95d6063
Sep 07 12:55:43.262561 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:43.262476    2565 generic.go:356] "Generic (PLEG): container finished" podID="539502f5-568f-4143-a9b5-e2a926f85a08" containerID="2b7bb6dc7e9791ab6f425f369cba3af0a1a2fc512ab989bc1f5437e87f406bd7" exitCode=0
Sep 07 12:55:43.262561 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:43.262529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"539502f5-568f-4143-a9b5-e2a926f85a08","Type":"ContainerDied","Data":"2b7bb6dc7e9791ab6f425f369cba3af0a1a2fc512ab989bc1f5437e87f406bd7"}
Sep 07 12:55:43.262954 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:43.262577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"539502f5-568f-4143-a9b5-e2a926f85a08","Type":"ContainerStarted","Data":"52dbd6c42bba2a2b15018f616c2a0b33700ee7d9962e46c0cbef10cdd95d6063"}
Sep 07 12:55:43.492858 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:43.492809    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ece4bc9d-7296-429a-9029-91b9f6c86689" path="/var/lib/kubelet/pods/ece4bc9d-7296-429a-9029-91b9f6c86689/volumes"
Sep 07 12:55:44.270379 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:44.270320    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"539502f5-568f-4143-a9b5-e2a926f85a08","Type":"ContainerStarted","Data":"ac2fccdeb194bcb3f9bfe22415d7561cd1801cb90ab65396f7c647eab723df2f"}
Sep 07 12:55:44.270862 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:44.270428    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"539502f5-568f-4143-a9b5-e2a926f85a08","Type":"ContainerStarted","Data":"c106c26737e956a87723debae3399e9568adcada098e846ec5e03d5826868ae0"}
Sep 07 12:55:44.270862 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:44.270471    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"539502f5-568f-4143-a9b5-e2a926f85a08","Type":"ContainerStarted","Data":"ce63c1bad72a674a4e8941a8886090c039453676431dc25824e0691d62d34d86"}
Sep 07 12:55:44.270862 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:44.270512    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"539502f5-568f-4143-a9b5-e2a926f85a08","Type":"ContainerStarted","Data":"8bf52ab3d0b7734e09111c43f506011ee18a29b0d8678eb4c3c535090a3f13f5"}
Sep 07 12:55:44.270862 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:44.270551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"539502f5-568f-4143-a9b5-e2a926f85a08","Type":"ContainerStarted","Data":"5bd703619349b3f11786f7e511c01bf00698bcf78a08bab2df06c7369f28328b"}
Sep 07 12:55:44.270862 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:44.270597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"539502f5-568f-4143-a9b5-e2a926f85a08","Type":"ContainerStarted","Data":"348af094a7bca3ab47eafdf62533266cdeaa6e5b6ee0ffd3a72a6ddb2777ae89"}
Sep 07 12:55:44.313479 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:44.313423    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.295393759 podStartE2EDuration="2.313405263s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.011504ms" isStatefulPod=false podCreationTimestamp="2026-09-07 12:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:55:44.308767464 +0000 UTC m=+161.339633786" watchObservedRunningTime="2026-09-07 12:55:44.313405263 +0000 UTC m=+161.344271572"
Sep 07 12:55:47.688593 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:55:47.688548    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:56:42.688401 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:56:42.688355    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:56:42.704361 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:56:42.704311    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:56:43.530525 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:56:43.530498    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 12:58:03.374987 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:03.374958    2565 kubelet.go:1648] "Image garbage collection succeeded"
Sep 07 12:58:27.479040 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:27.479000    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ffrq5"]
Sep 07 12:58:27.482369 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:27.482347    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ffrq5"
Sep 07 12:58:27.484839 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:27.484814    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 07 12:58:27.484968 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:27.484916    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-plcgk\""
Sep 07 12:58:27.485719 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:27.485699    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 07 12:58:27.500494 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:27.500468    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ffrq5"]
Sep 07 12:58:27.596210 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:27.596172    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs764\" (UniqueName: \"kubernetes.io/projected/7da774c2-b32b-4364-9314-0b5d7b029225-kube-api-access-cs764\") pod \"cert-manager-cainjector-559cdd8c87-ffrq5\" (UID: \"7da774c2-b32b-4364-9314-0b5d7b029225\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ffrq5"
Sep 07 12:58:27.596419 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:27.596232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7da774c2-b32b-4364-9314-0b5d7b029225-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ffrq5\" (UID: \"7da774c2-b32b-4364-9314-0b5d7b029225\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ffrq5"
Sep 07 12:58:27.697252 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:27.697210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs764\" (UniqueName: \"kubernetes.io/projected/7da774c2-b32b-4364-9314-0b5d7b029225-kube-api-access-cs764\") pod \"cert-manager-cainjector-559cdd8c87-ffrq5\" (UID: \"7da774c2-b32b-4364-9314-0b5d7b029225\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ffrq5"
Sep 07 12:58:27.697448 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:27.697271    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7da774c2-b32b-4364-9314-0b5d7b029225-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ffrq5\" (UID: \"7da774c2-b32b-4364-9314-0b5d7b029225\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ffrq5"
Sep 07 12:58:27.707501 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:27.707477    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7da774c2-b32b-4364-9314-0b5d7b029225-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ffrq5\" (UID: \"7da774c2-b32b-4364-9314-0b5d7b029225\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ffrq5"
Sep 07 12:58:27.707837 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:27.707813    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs764\" (UniqueName: \"kubernetes.io/projected/7da774c2-b32b-4364-9314-0b5d7b029225-kube-api-access-cs764\") pod \"cert-manager-cainjector-559cdd8c87-ffrq5\" (UID: \"7da774c2-b32b-4364-9314-0b5d7b029225\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ffrq5"
Sep 07 12:58:27.805114 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:27.805083    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ffrq5"
Sep 07 12:58:28.082647 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:28.082623    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ffrq5"]
Sep 07 12:58:28.085438 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:58:28.085403    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7da774c2_b32b_4364_9314_0b5d7b029225.slice/crio-a8f73835bf13de4959a8d0871bfa938f31bbead084736c7fed388ef843edad46 WatchSource:0}: Error finding container a8f73835bf13de4959a8d0871bfa938f31bbead084736c7fed388ef843edad46: Status 404 returned error can't find the container with id a8f73835bf13de4959a8d0871bfa938f31bbead084736c7fed388ef843edad46
Sep 07 12:58:28.087525 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:28.087502    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 12:58:28.932906 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:28.932867    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ffrq5" event={"ID":"7da774c2-b32b-4364-9314-0b5d7b029225","Type":"ContainerStarted","Data":"a8f73835bf13de4959a8d0871bfa938f31bbead084736c7fed388ef843edad46"}
Sep 07 12:58:31.953051 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:31.953011    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ffrq5" event={"ID":"7da774c2-b32b-4364-9314-0b5d7b029225","Type":"ContainerStarted","Data":"3130a176b4d45d2b0f41da27a9e6f00a26213b6cd5ee97513dab2886634e834d"}
Sep 07 12:58:31.970690 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:31.970634    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ffrq5" podStartSLOduration=1.982963182 podStartE2EDuration="4.970618178s" totalImagesPullingTime="2.987654996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:58:31.969195575 +0000 UTC m=+329.000061887" watchObservedRunningTime="2026-09-07 12:58:31.970618178 +0000 UTC m=+329.001484488"
Sep 07 12:58:36.393521 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:36.393470    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-g4nfs"]
Sep 07 12:58:36.396851 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:36.396831    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-g4nfs"
Sep 07 12:58:36.399290 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:36.399266    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-pmmxt\""
Sep 07 12:58:36.408070 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:36.408048    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-g4nfs"]
Sep 07 12:58:36.473010 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:36.472973    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxmpd\" (UniqueName: \"kubernetes.io/projected/daa8d793-0e10-4530-9509-8c9f88127e32-kube-api-access-fxmpd\") pod \"cert-manager-55b4d954d5-g4nfs\" (UID: \"daa8d793-0e10-4530-9509-8c9f88127e32\") " pod="cert-manager/cert-manager-55b4d954d5-g4nfs"
Sep 07 12:58:36.473010 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:36.473015    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/daa8d793-0e10-4530-9509-8c9f88127e32-bound-sa-token\") pod \"cert-manager-55b4d954d5-g4nfs\" (UID: \"daa8d793-0e10-4530-9509-8c9f88127e32\") " pod="cert-manager/cert-manager-55b4d954d5-g4nfs"
Sep 07 12:58:36.573568 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:36.573534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxmpd\" (UniqueName: \"kubernetes.io/projected/daa8d793-0e10-4530-9509-8c9f88127e32-kube-api-access-fxmpd\") pod \"cert-manager-55b4d954d5-g4nfs\" (UID: \"daa8d793-0e10-4530-9509-8c9f88127e32\") " pod="cert-manager/cert-manager-55b4d954d5-g4nfs"
Sep 07 12:58:36.573568 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:36.573575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/daa8d793-0e10-4530-9509-8c9f88127e32-bound-sa-token\") pod \"cert-manager-55b4d954d5-g4nfs\" (UID: \"daa8d793-0e10-4530-9509-8c9f88127e32\") " pod="cert-manager/cert-manager-55b4d954d5-g4nfs"
Sep 07 12:58:36.584610 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:36.584572    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/daa8d793-0e10-4530-9509-8c9f88127e32-bound-sa-token\") pod \"cert-manager-55b4d954d5-g4nfs\" (UID: \"daa8d793-0e10-4530-9509-8c9f88127e32\") " pod="cert-manager/cert-manager-55b4d954d5-g4nfs"
Sep 07 12:58:36.584727 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:36.584713    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxmpd\" (UniqueName: \"kubernetes.io/projected/daa8d793-0e10-4530-9509-8c9f88127e32-kube-api-access-fxmpd\") pod \"cert-manager-55b4d954d5-g4nfs\" (UID: \"daa8d793-0e10-4530-9509-8c9f88127e32\") " pod="cert-manager/cert-manager-55b4d954d5-g4nfs"
Sep 07 12:58:36.706018 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:36.705916    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-g4nfs"
Sep 07 12:58:36.986707 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:36.986680    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-g4nfs"]
Sep 07 12:58:36.988884 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:58:36.988849    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddaa8d793_0e10_4530_9509_8c9f88127e32.slice/crio-bf9b41d4044fd5874dcf420095c9d568c6e1eafb64150c1b2ce99979cd62bc86 WatchSource:0}: Error finding container bf9b41d4044fd5874dcf420095c9d568c6e1eafb64150c1b2ce99979cd62bc86: Status 404 returned error can't find the container with id bf9b41d4044fd5874dcf420095c9d568c6e1eafb64150c1b2ce99979cd62bc86
Sep 07 12:58:37.978946 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:37.978907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-g4nfs" event={"ID":"daa8d793-0e10-4530-9509-8c9f88127e32","Type":"ContainerStarted","Data":"c911994d9994220a509665a0629169483a8c3438926970fc4244736ddac12c58"}
Sep 07 12:58:37.978946 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:37.978952    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-g4nfs" event={"ID":"daa8d793-0e10-4530-9509-8c9f88127e32","Type":"ContainerStarted","Data":"bf9b41d4044fd5874dcf420095c9d568c6e1eafb64150c1b2ce99979cd62bc86"}
Sep 07 12:58:37.997253 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:58:37.997210    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-g4nfs" podStartSLOduration=1.997194007 podStartE2EDuration="1.997194007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:58:37.995695153 +0000 UTC m=+335.026561457" watchObservedRunningTime="2026-09-07 12:58:37.997194007 +0000 UTC m=+335.028060317"
Sep 07 12:59:12.702262 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.702187    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"]
Sep 07 12:59:12.705700 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.705680    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"
Sep 07 12:59:12.708092 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.708071    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 07 12:59:12.708907 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.708860    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8kkcd\""
Sep 07 12:59:12.709027 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.708902    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 07 12:59:12.709027 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.708917    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 07 12:59:12.709162 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.709109    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 07 12:59:12.709229 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.709188    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 07 12:59:12.714413 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.714393    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"]
Sep 07 12:59:12.893272 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.893237    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szw8m\" (UniqueName: \"kubernetes.io/projected/f0662053-a65e-4df5-b54d-c98d9d968cff-kube-api-access-szw8m\") pod \"lws-controller-manager-78895d9f88-lwjfq\" (UID: \"f0662053-a65e-4df5-b54d-c98d9d968cff\") " pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"
Sep 07 12:59:12.893272 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.893278    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0662053-a65e-4df5-b54d-c98d9d968cff-metrics-cert\") pod \"lws-controller-manager-78895d9f88-lwjfq\" (UID: \"f0662053-a65e-4df5-b54d-c98d9d968cff\") " pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"
Sep 07 12:59:12.893515 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.893297    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f0662053-a65e-4df5-b54d-c98d9d968cff-manager-config\") pod \"lws-controller-manager-78895d9f88-lwjfq\" (UID: \"f0662053-a65e-4df5-b54d-c98d9d968cff\") " pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"
Sep 07 12:59:12.893515 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.893408    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0662053-a65e-4df5-b54d-c98d9d968cff-cert\") pod \"lws-controller-manager-78895d9f88-lwjfq\" (UID: \"f0662053-a65e-4df5-b54d-c98d9d968cff\") " pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"
Sep 07 12:59:12.994145 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.994067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0662053-a65e-4df5-b54d-c98d9d968cff-cert\") pod \"lws-controller-manager-78895d9f88-lwjfq\" (UID: \"f0662053-a65e-4df5-b54d-c98d9d968cff\") " pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"
Sep 07 12:59:12.994145 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.994128    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szw8m\" (UniqueName: \"kubernetes.io/projected/f0662053-a65e-4df5-b54d-c98d9d968cff-kube-api-access-szw8m\") pod \"lws-controller-manager-78895d9f88-lwjfq\" (UID: \"f0662053-a65e-4df5-b54d-c98d9d968cff\") " pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"
Sep 07 12:59:12.994145 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.994148    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0662053-a65e-4df5-b54d-c98d9d968cff-metrics-cert\") pod \"lws-controller-manager-78895d9f88-lwjfq\" (UID: \"f0662053-a65e-4df5-b54d-c98d9d968cff\") " pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"
Sep 07 12:59:12.994475 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.994165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f0662053-a65e-4df5-b54d-c98d9d968cff-manager-config\") pod \"lws-controller-manager-78895d9f88-lwjfq\" (UID: \"f0662053-a65e-4df5-b54d-c98d9d968cff\") " pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"
Sep 07 12:59:12.994915 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.994891    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f0662053-a65e-4df5-b54d-c98d9d968cff-manager-config\") pod \"lws-controller-manager-78895d9f88-lwjfq\" (UID: \"f0662053-a65e-4df5-b54d-c98d9d968cff\") " pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"
Sep 07 12:59:12.996736 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.996707    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0662053-a65e-4df5-b54d-c98d9d968cff-metrics-cert\") pod \"lws-controller-manager-78895d9f88-lwjfq\" (UID: \"f0662053-a65e-4df5-b54d-c98d9d968cff\") " pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"
Sep 07 12:59:12.996736 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:12.996731    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0662053-a65e-4df5-b54d-c98d9d968cff-cert\") pod \"lws-controller-manager-78895d9f88-lwjfq\" (UID: \"f0662053-a65e-4df5-b54d-c98d9d968cff\") " pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"
Sep 07 12:59:13.014264 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:13.014233    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szw8m\" (UniqueName: \"kubernetes.io/projected/f0662053-a65e-4df5-b54d-c98d9d968cff-kube-api-access-szw8m\") pod \"lws-controller-manager-78895d9f88-lwjfq\" (UID: \"f0662053-a65e-4df5-b54d-c98d9d968cff\") " pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"
Sep 07 12:59:13.014966 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:13.014945    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"
Sep 07 12:59:13.301761 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:13.301726    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"]
Sep 07 12:59:13.304791 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:59:13.304745    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0662053_a65e_4df5_b54d_c98d9d968cff.slice/crio-94c8b62e38055c4d2eb5b52572c5038936ef40da9f0db98242b5404224bbdbf0 WatchSource:0}: Error finding container 94c8b62e38055c4d2eb5b52572c5038936ef40da9f0db98242b5404224bbdbf0: Status 404 returned error can't find the container with id 94c8b62e38055c4d2eb5b52572c5038936ef40da9f0db98242b5404224bbdbf0
Sep 07 12:59:14.127645 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:14.127612    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq" event={"ID":"f0662053-a65e-4df5-b54d-c98d9d968cff","Type":"ContainerStarted","Data":"94c8b62e38055c4d2eb5b52572c5038936ef40da9f0db98242b5404224bbdbf0"}
Sep 07 12:59:17.142351 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:17.142288    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq" event={"ID":"f0662053-a65e-4df5-b54d-c98d9d968cff","Type":"ContainerStarted","Data":"81fbb216132114af5417737c40d5efae030944a7d953be210493bbeb684fe8fd"}
Sep 07 12:59:17.142759 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:17.142407    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"
Sep 07 12:59:17.164215 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:17.164162    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq" podStartSLOduration=2.187751209 podStartE2EDuration="5.164146804s" totalImagesPullingTime="2.976395595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:17.161790626 +0000 UTC m=+374.192656935" watchObservedRunningTime="2026-09-07 12:59:17.164146804 +0000 UTC m=+374.195013114"
Sep 07 12:59:21.148724 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.148692    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"]
Sep 07 12:59:21.152724 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.152696    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.155255 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.155220    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/182dd5d3-d39d-4aa1-9708-a912ad59a5f7-webhook-cert\") pod \"opendatahub-operator-controller-manager-cc4b9478f-md9pt\" (UID: \"182dd5d3-d39d-4aa1-9708-a912ad59a5f7\") " pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.155402 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.155344    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/182dd5d3-d39d-4aa1-9708-a912ad59a5f7-manifests\") pod \"opendatahub-operator-controller-manager-cc4b9478f-md9pt\" (UID: \"182dd5d3-d39d-4aa1-9708-a912ad59a5f7\") " pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.155402 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.155378    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/182dd5d3-d39d-4aa1-9708-a912ad59a5f7-tmp\") pod \"opendatahub-operator-controller-manager-cc4b9478f-md9pt\" (UID: \"182dd5d3-d39d-4aa1-9708-a912ad59a5f7\") " pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.155507 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.155428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/182dd5d3-d39d-4aa1-9708-a912ad59a5f7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cc4b9478f-md9pt\" (UID: \"182dd5d3-d39d-4aa1-9708-a912ad59a5f7\") " pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.155507 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.155495    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2tlm\" (UniqueName: \"kubernetes.io/projected/182dd5d3-d39d-4aa1-9708-a912ad59a5f7-kube-api-access-x2tlm\") pod \"opendatahub-operator-controller-manager-cc4b9478f-md9pt\" (UID: \"182dd5d3-d39d-4aa1-9708-a912ad59a5f7\") " pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.155938 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.155913    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 07 12:59:21.156104 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.156078    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 07 12:59:21.156195 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.156176    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hz4rq\""
Sep 07 12:59:21.156481 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.156461    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 07 12:59:21.156921 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.156897    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 07 12:59:21.169741 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.169717    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"]
Sep 07 12:59:21.255900 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.255861    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/182dd5d3-d39d-4aa1-9708-a912ad59a5f7-manifests\") pod \"opendatahub-operator-controller-manager-cc4b9478f-md9pt\" (UID: \"182dd5d3-d39d-4aa1-9708-a912ad59a5f7\") " pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.255900 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.255897    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/182dd5d3-d39d-4aa1-9708-a912ad59a5f7-tmp\") pod \"opendatahub-operator-controller-manager-cc4b9478f-md9pt\" (UID: \"182dd5d3-d39d-4aa1-9708-a912ad59a5f7\") " pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.256155 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.255922    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/182dd5d3-d39d-4aa1-9708-a912ad59a5f7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cc4b9478f-md9pt\" (UID: \"182dd5d3-d39d-4aa1-9708-a912ad59a5f7\") " pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.256155 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.255973    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tlm\" (UniqueName: \"kubernetes.io/projected/182dd5d3-d39d-4aa1-9708-a912ad59a5f7-kube-api-access-x2tlm\") pod \"opendatahub-operator-controller-manager-cc4b9478f-md9pt\" (UID: \"182dd5d3-d39d-4aa1-9708-a912ad59a5f7\") " pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.256155 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.256015    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/182dd5d3-d39d-4aa1-9708-a912ad59a5f7-webhook-cert\") pod \"opendatahub-operator-controller-manager-cc4b9478f-md9pt\" (UID: \"182dd5d3-d39d-4aa1-9708-a912ad59a5f7\") " pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.256383 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.256354    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/182dd5d3-d39d-4aa1-9708-a912ad59a5f7-manifests\") pod \"opendatahub-operator-controller-manager-cc4b9478f-md9pt\" (UID: \"182dd5d3-d39d-4aa1-9708-a912ad59a5f7\") " pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.258528 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.258497    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/182dd5d3-d39d-4aa1-9708-a912ad59a5f7-tmp\") pod \"opendatahub-operator-controller-manager-cc4b9478f-md9pt\" (UID: \"182dd5d3-d39d-4aa1-9708-a912ad59a5f7\") " pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.258655 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.258625    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/182dd5d3-d39d-4aa1-9708-a912ad59a5f7-webhook-cert\") pod \"opendatahub-operator-controller-manager-cc4b9478f-md9pt\" (UID: \"182dd5d3-d39d-4aa1-9708-a912ad59a5f7\") " pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.258751 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.258728    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/182dd5d3-d39d-4aa1-9708-a912ad59a5f7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cc4b9478f-md9pt\" (UID: \"182dd5d3-d39d-4aa1-9708-a912ad59a5f7\") " pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.267793 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.267764    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2tlm\" (UniqueName: \"kubernetes.io/projected/182dd5d3-d39d-4aa1-9708-a912ad59a5f7-kube-api-access-x2tlm\") pod \"opendatahub-operator-controller-manager-cc4b9478f-md9pt\" (UID: \"182dd5d3-d39d-4aa1-9708-a912ad59a5f7\") " pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.470521 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.470431    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:21.757460 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:21.757377    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"]
Sep 07 12:59:21.761560 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:59:21.761521    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182dd5d3_d39d_4aa1_9708_a912ad59a5f7.slice/crio-3ec6eba937a92de9d94e1e1f16d091faec3f9505451adc4d963ebdf34951b534 WatchSource:0}: Error finding container 3ec6eba937a92de9d94e1e1f16d091faec3f9505451adc4d963ebdf34951b534: Status 404 returned error can't find the container with id 3ec6eba937a92de9d94e1e1f16d091faec3f9505451adc4d963ebdf34951b534
Sep 07 12:59:22.168344 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:22.168306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt" event={"ID":"182dd5d3-d39d-4aa1-9708-a912ad59a5f7","Type":"ContainerStarted","Data":"3ec6eba937a92de9d94e1e1f16d091faec3f9505451adc4d963ebdf34951b534"}
Sep 07 12:59:25.185984 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:25.185939    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt" event={"ID":"182dd5d3-d39d-4aa1-9708-a912ad59a5f7","Type":"ContainerStarted","Data":"727c63d828378fb9b1ebfdca3af8876a4d36b52bf897b10b6d40a4e268779ef6"}
Sep 07 12:59:26.192393 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:26.192354    2565 generic.go:356] "Generic (PLEG): container finished" podID="182dd5d3-d39d-4aa1-9708-a912ad59a5f7" containerID="727c63d828378fb9b1ebfdca3af8876a4d36b52bf897b10b6d40a4e268779ef6" exitCode=0
Sep 07 12:59:26.192843 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:26.192448    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt" event={"ID":"182dd5d3-d39d-4aa1-9708-a912ad59a5f7","Type":"ContainerDied","Data":"727c63d828378fb9b1ebfdca3af8876a4d36b52bf897b10b6d40a4e268779ef6"}
Sep 07 12:59:27.200982 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:27.200942    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt" event={"ID":"182dd5d3-d39d-4aa1-9708-a912ad59a5f7","Type":"ContainerStarted","Data":"e9f121e98ac08b94f707178fad54a1da9ea07840957fb7a232f40ae9e9f026a0"}
Sep 07 12:59:27.201492 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:27.201048    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:27.226666 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:27.226603    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt" podStartSLOduration=1.9226235919999999 podStartE2EDuration="6.226584203s" totalImagesPullingTime="3.246075998s" totalInitContainerRuntime="1.057884613s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:59:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:27.225304564 +0000 UTC m=+384.256170873" watchObservedRunningTime="2026-09-07 12:59:27.226584203 +0000 UTC m=+384.257450514"
Sep 07 12:59:28.149254 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:28.149227    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-78895d9f88-lwjfq"
Sep 07 12:59:38.210069 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:38.210038    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-cc4b9478f-md9pt"
Sep 07 12:59:42.370542 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.370503    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz"]
Sep 07 12:59:42.387519 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.387483    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz"]
Sep 07 12:59:42.387710 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.387629    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz"
Sep 07 12:59:42.390607 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.390583    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 07 12:59:42.390785 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.390760    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 07 12:59:42.390901 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.390594    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 07 12:59:42.391137 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.391117    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4wctm\""
Sep 07 12:59:42.391369 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.391349    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 07 12:59:42.429889 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.429852    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/354de67c-b720-402c-9307-6d8467a103b2-tmp\") pod \"kube-auth-proxy-558c85b7d6-q7pcz\" (UID: \"354de67c-b720-402c-9307-6d8467a103b2\") " pod="openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz"
Sep 07 12:59:42.430035 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.429958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvk5f\" (UniqueName: \"kubernetes.io/projected/354de67c-b720-402c-9307-6d8467a103b2-kube-api-access-fvk5f\") pod \"kube-auth-proxy-558c85b7d6-q7pcz\" (UID: \"354de67c-b720-402c-9307-6d8467a103b2\") " pod="openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz"
Sep 07 12:59:42.430035 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.430014    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354de67c-b720-402c-9307-6d8467a103b2-tls-certs\") pod \"kube-auth-proxy-558c85b7d6-q7pcz\" (UID: \"354de67c-b720-402c-9307-6d8467a103b2\") " pod="openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz"
Sep 07 12:59:42.531150 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.531106    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvk5f\" (UniqueName: \"kubernetes.io/projected/354de67c-b720-402c-9307-6d8467a103b2-kube-api-access-fvk5f\") pod \"kube-auth-proxy-558c85b7d6-q7pcz\" (UID: \"354de67c-b720-402c-9307-6d8467a103b2\") " pod="openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz"
Sep 07 12:59:42.531369 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.531173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354de67c-b720-402c-9307-6d8467a103b2-tls-certs\") pod \"kube-auth-proxy-558c85b7d6-q7pcz\" (UID: \"354de67c-b720-402c-9307-6d8467a103b2\") " pod="openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz"
Sep 07 12:59:42.531369 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.531230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/354de67c-b720-402c-9307-6d8467a103b2-tmp\") pod \"kube-auth-proxy-558c85b7d6-q7pcz\" (UID: \"354de67c-b720-402c-9307-6d8467a103b2\") " pod="openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz"
Sep 07 12:59:42.533763 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.533737    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/354de67c-b720-402c-9307-6d8467a103b2-tmp\") pod \"kube-auth-proxy-558c85b7d6-q7pcz\" (UID: \"354de67c-b720-402c-9307-6d8467a103b2\") " pod="openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz"
Sep 07 12:59:42.533957 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.533942    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354de67c-b720-402c-9307-6d8467a103b2-tls-certs\") pod \"kube-auth-proxy-558c85b7d6-q7pcz\" (UID: \"354de67c-b720-402c-9307-6d8467a103b2\") " pod="openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz"
Sep 07 12:59:42.541288 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.541265    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvk5f\" (UniqueName: \"kubernetes.io/projected/354de67c-b720-402c-9307-6d8467a103b2-kube-api-access-fvk5f\") pod \"kube-auth-proxy-558c85b7d6-q7pcz\" (UID: \"354de67c-b720-402c-9307-6d8467a103b2\") " pod="openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz"
Sep 07 12:59:42.698175 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.698102    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz"
Sep 07 12:59:42.981776 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:42.981695    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz"]
Sep 07 12:59:42.985877 ip-10-0-141-58 kubenswrapper[2565]: W0907 12:59:42.985829    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354de67c_b720_402c_9307_6d8467a103b2.slice/crio-9c1a243be4154668043a7c065c3e8b66b36afb09bcf4e7cac627590daaecb266 WatchSource:0}: Error finding container 9c1a243be4154668043a7c065c3e8b66b36afb09bcf4e7cac627590daaecb266: Status 404 returned error can't find the container with id 9c1a243be4154668043a7c065c3e8b66b36afb09bcf4e7cac627590daaecb266
Sep 07 12:59:43.277521 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:43.277438    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz" event={"ID":"354de67c-b720-402c-9307-6d8467a103b2","Type":"ContainerStarted","Data":"9c1a243be4154668043a7c065c3e8b66b36afb09bcf4e7cac627590daaecb266"}
Sep 07 12:59:46.297377 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:46.297344    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz" event={"ID":"354de67c-b720-402c-9307-6d8467a103b2","Type":"ContainerStarted","Data":"1a8296d33a2efc77680e92919c28d371f91a4e64aa3f5c71dc83eca834806fb2"}
Sep 07 12:59:46.316266 ip-10-0-141-58 kubenswrapper[2565]: I0907 12:59:46.316214    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-558c85b7d6-q7pcz" podStartSLOduration=1.457396529 podStartE2EDuration="4.316199381s" totalImagesPullingTime="2.858802852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:46.315319317 +0000 UTC m=+403.346185626" watchObservedRunningTime="2026-09-07 12:59:46.316199381 +0000 UTC m=+403.347065691"
Sep 07 13:02:08.499287 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.499205    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"]
Sep 07 13:02:08.502802 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.502784    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.505203 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.505185    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rt24r\""
Sep 07 13:02:08.505307 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.505188    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 07 13:02:08.519208 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.519184    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"]
Sep 07 13:02:08.552078 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.552050    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fad632-9c22-4303-8c09-aa25f954ccaa-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.552209 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.552134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08fad632-9c22-4303-8c09-aa25f954ccaa-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.552209 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.552160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08fad632-9c22-4303-8c09-aa25f954ccaa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.552209 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.552191    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08fad632-9c22-4303-8c09-aa25f954ccaa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.552316 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.552219    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zffmb\" (UniqueName: \"kubernetes.io/projected/08fad632-9c22-4303-8c09-aa25f954ccaa-kube-api-access-zffmb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.552316 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.552267    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08fad632-9c22-4303-8c09-aa25f954ccaa-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.552316 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.552296    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08fad632-9c22-4303-8c09-aa25f954ccaa-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.552450 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.552347    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fad632-9c22-4303-8c09-aa25f954ccaa-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.552450 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.552364    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08fad632-9c22-4303-8c09-aa25f954ccaa-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.653262 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.653230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08fad632-9c22-4303-8c09-aa25f954ccaa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.653262 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.653267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zffmb\" (UniqueName: \"kubernetes.io/projected/08fad632-9c22-4303-8c09-aa25f954ccaa-kube-api-access-zffmb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.653537 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.653295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08fad632-9c22-4303-8c09-aa25f954ccaa-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.653537 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.653362    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08fad632-9c22-4303-8c09-aa25f954ccaa-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.653537 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.653397    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fad632-9c22-4303-8c09-aa25f954ccaa-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.653537 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.653430    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08fad632-9c22-4303-8c09-aa25f954ccaa-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.653537 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.653471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fad632-9c22-4303-8c09-aa25f954ccaa-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.653791 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.653568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08fad632-9c22-4303-8c09-aa25f954ccaa-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.653791 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.653594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08fad632-9c22-4303-8c09-aa25f954ccaa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.653936 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.653911    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fad632-9c22-4303-8c09-aa25f954ccaa-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.653990 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.653961    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08fad632-9c22-4303-8c09-aa25f954ccaa-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.654042 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.653985    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fad632-9c22-4303-8c09-aa25f954ccaa-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.654135 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.654090    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08fad632-9c22-4303-8c09-aa25f954ccaa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.654356 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.654193    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08fad632-9c22-4303-8c09-aa25f954ccaa-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.655861 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.655841    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08fad632-9c22-4303-8c09-aa25f954ccaa-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.656199 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.656177    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08fad632-9c22-4303-8c09-aa25f954ccaa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.662542 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.662520    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08fad632-9c22-4303-8c09-aa25f954ccaa-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.662838 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.662815    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zffmb\" (UniqueName: \"kubernetes.io/projected/08fad632-9c22-4303-8c09-aa25f954ccaa-kube-api-access-zffmb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pstv4\" (UID: \"08fad632-9c22-4303-8c09-aa25f954ccaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:08.813188 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:08.813154    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:09.095294 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:09.095234    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"]
Sep 07 13:02:09.097778 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:02:09.097748    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fad632_9c22_4303_8c09_aa25f954ccaa.slice/crio-1e5823e3b1e9a46d2e1e268ba52eb96e40d18bdad2d6ae39bf567215fa04a625 WatchSource:0}: Error finding container 1e5823e3b1e9a46d2e1e268ba52eb96e40d18bdad2d6ae39bf567215fa04a625: Status 404 returned error can't find the container with id 1e5823e3b1e9a46d2e1e268ba52eb96e40d18bdad2d6ae39bf567215fa04a625
Sep 07 13:02:09.963785 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:09.963751    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4" event={"ID":"08fad632-9c22-4303-8c09-aa25f954ccaa","Type":"ContainerStarted","Data":"1e5823e3b1e9a46d2e1e268ba52eb96e40d18bdad2d6ae39bf567215fa04a625"}
Sep 07 13:02:11.894398 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:11.894348    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:02:11.894823 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:11.894448    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:02:11.894823 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:11.894495    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:02:11.977725 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:11.977690    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4" event={"ID":"08fad632-9c22-4303-8c09-aa25f954ccaa","Type":"ContainerStarted","Data":"e5b3805007d636cc1b4ef736498431e8cbf86f8a275de33c25a18e376e44561b"}
Sep 07 13:02:12.001367 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:12.001295    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4" podStartSLOduration=1.2076359939999999 podStartE2EDuration="4.001279892s" totalImagesPullingTime="2.793643898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:02:12.000136675 +0000 UTC m=+549.031003015" watchObservedRunningTime="2026-09-07 13:02:12.001279892 +0000 UTC m=+549.032146201"
Sep 07 13:02:12.813827 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:12.813797    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:12.818669 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:12.818635    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:12.983493 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:12.983467    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:12.984391 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:12.984373    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:02:23.433880 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:23.433845    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:02:23.437448 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:23.437427    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"
Sep 07 13:02:23.439800 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:23.439782    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7c2jp\""
Sep 07 13:02:23.439899 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:23.439879    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 07 13:02:23.439964 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:23.439879    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 07 13:02:23.440819 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:23.440805    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 07 13:02:23.451659 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:23.451638    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:02:23.479793 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:23.479764    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:02:23.594388 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:23.594348    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eba692f7-75ab-487a-94c0-6e90e89ea54c-config-file\") pod \"limitador-limitador-85d9db4d5d-z2nx8\" (UID: \"eba692f7-75ab-487a-94c0-6e90e89ea54c\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"
Sep 07 13:02:23.594570 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:23.594409    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7glxj\" (UniqueName: \"kubernetes.io/projected/eba692f7-75ab-487a-94c0-6e90e89ea54c-kube-api-access-7glxj\") pod \"limitador-limitador-85d9db4d5d-z2nx8\" (UID: \"eba692f7-75ab-487a-94c0-6e90e89ea54c\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"
Sep 07 13:02:23.695758 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:23.695667    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eba692f7-75ab-487a-94c0-6e90e89ea54c-config-file\") pod \"limitador-limitador-85d9db4d5d-z2nx8\" (UID: \"eba692f7-75ab-487a-94c0-6e90e89ea54c\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"
Sep 07 13:02:23.695758 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:23.695725    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7glxj\" (UniqueName: \"kubernetes.io/projected/eba692f7-75ab-487a-94c0-6e90e89ea54c-kube-api-access-7glxj\") pod \"limitador-limitador-85d9db4d5d-z2nx8\" (UID: \"eba692f7-75ab-487a-94c0-6e90e89ea54c\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"
Sep 07 13:02:23.696310 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:23.696285    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eba692f7-75ab-487a-94c0-6e90e89ea54c-config-file\") pod \"limitador-limitador-85d9db4d5d-z2nx8\" (UID: \"eba692f7-75ab-487a-94c0-6e90e89ea54c\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"
Sep 07 13:02:23.706054 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:23.706030    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7glxj\" (UniqueName: \"kubernetes.io/projected/eba692f7-75ab-487a-94c0-6e90e89ea54c-kube-api-access-7glxj\") pod \"limitador-limitador-85d9db4d5d-z2nx8\" (UID: \"eba692f7-75ab-487a-94c0-6e90e89ea54c\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"
Sep 07 13:02:23.747281 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:23.747245    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"
Sep 07 13:02:24.023186 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:24.023156    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:02:24.025689 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:02:24.025659    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba692f7_75ab_487a_94c0_6e90e89ea54c.slice/crio-6d8c3e10ce8c13ac659e4739965fe6b275db8569b74cc10fc2e4bc9dc49c7305 WatchSource:0}: Error finding container 6d8c3e10ce8c13ac659e4739965fe6b275db8569b74cc10fc2e4bc9dc49c7305: Status 404 returned error can't find the container with id 6d8c3e10ce8c13ac659e4739965fe6b275db8569b74cc10fc2e4bc9dc49c7305
Sep 07 13:02:24.041610 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:24.041582    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8" event={"ID":"eba692f7-75ab-487a-94c0-6e90e89ea54c","Type":"ContainerStarted","Data":"6d8c3e10ce8c13ac659e4739965fe6b275db8569b74cc10fc2e4bc9dc49c7305"}
Sep 07 13:02:27.057438 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:27.057402    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8" event={"ID":"eba692f7-75ab-487a-94c0-6e90e89ea54c","Type":"ContainerStarted","Data":"785ebf6ae35a82ac1a3f9a3e0e5d5f380b488defc567b1a7338a8f910b411f6d"}
Sep 07 13:02:27.057881 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:27.057565    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"
Sep 07 13:02:27.081464 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:27.081414    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8" podStartSLOduration=1.43264695 podStartE2EDuration="4.081398058s" totalImagesPullingTime="2.648751108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:02:27.080003708 +0000 UTC m=+564.110870020" watchObservedRunningTime="2026-09-07 13:02:27.081398058 +0000 UTC m=+564.112264367"
Sep 07 13:02:38.062738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:02:38.062709    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"
Sep 07 13:03:39.506040 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:39.506007    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-gqz94"]
Sep 07 13:03:39.509214 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:39.509199    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-gqz94"
Sep 07 13:03:39.511977 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:39.511952    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 07 13:03:39.512458 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:39.512438    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-46nsv\""
Sep 07 13:03:39.525111 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:39.525089    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-gqz94"]
Sep 07 13:03:39.582062 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:39.582027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338d8b09-0346-4935-8010-32425a4c3de7-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-gqz94\" (UID: \"338d8b09-0346-4935-8010-32425a4c3de7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-gqz94"
Sep 07 13:03:39.582211 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:39.582088    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx6hf\" (UniqueName: \"kubernetes.io/projected/338d8b09-0346-4935-8010-32425a4c3de7-kube-api-access-wx6hf\") pod \"payload-pre-processing-59fbc47bf8-gqz94\" (UID: \"338d8b09-0346-4935-8010-32425a4c3de7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-gqz94"
Sep 07 13:03:39.610252 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:39.610221    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:03:39.683070 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:39.683033    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338d8b09-0346-4935-8010-32425a4c3de7-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-gqz94\" (UID: \"338d8b09-0346-4935-8010-32425a4c3de7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-gqz94"
Sep 07 13:03:39.683239 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:39.683112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx6hf\" (UniqueName: \"kubernetes.io/projected/338d8b09-0346-4935-8010-32425a4c3de7-kube-api-access-wx6hf\") pod \"payload-pre-processing-59fbc47bf8-gqz94\" (UID: \"338d8b09-0346-4935-8010-32425a4c3de7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-gqz94"
Sep 07 13:03:39.683786 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:39.683766    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/338d8b09-0346-4935-8010-32425a4c3de7-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-gqz94\" (UID: \"338d8b09-0346-4935-8010-32425a4c3de7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-gqz94"
Sep 07 13:03:39.694194 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:39.694168    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx6hf\" (UniqueName: \"kubernetes.io/projected/338d8b09-0346-4935-8010-32425a4c3de7-kube-api-access-wx6hf\") pod \"payload-pre-processing-59fbc47bf8-gqz94\" (UID: \"338d8b09-0346-4935-8010-32425a4c3de7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-gqz94"
Sep 07 13:03:39.819542 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:39.819510    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-gqz94"
Sep 07 13:03:40.100733 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:40.100652    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-gqz94"]
Sep 07 13:03:40.104252 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:03:40.104223    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338d8b09_0346_4935_8010_32425a4c3de7.slice/crio-20f28bd38eebea71dcd6587924fc7312da037c7d335e4b96e28a526d101384ad WatchSource:0}: Error finding container 20f28bd38eebea71dcd6587924fc7312da037c7d335e4b96e28a526d101384ad: Status 404 returned error can't find the container with id 20f28bd38eebea71dcd6587924fc7312da037c7d335e4b96e28a526d101384ad
Sep 07 13:03:40.106162 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:40.106147    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 13:03:40.402084 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:40.401987    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-gqz94" event={"ID":"338d8b09-0346-4935-8010-32425a4c3de7","Type":"ContainerStarted","Data":"20f28bd38eebea71dcd6587924fc7312da037c7d335e4b96e28a526d101384ad"}
Sep 07 13:03:42.416400 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:42.416364    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-gqz94" event={"ID":"338d8b09-0346-4935-8010-32425a4c3de7","Type":"ContainerStarted","Data":"d3c12b8fc7c332677a6ab271d92c739848df410701b6a65670538ebd11ac58eb"}
Sep 07 13:03:42.416801 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:42.416468    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-gqz94"
Sep 07 13:03:42.440373 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:42.440302    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-gqz94" podStartSLOduration=1.623669681 podStartE2EDuration="3.440287355s" totalImagesPullingTime="1.816617674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:03:42.437966426 +0000 UTC m=+639.468832737" watchObservedRunningTime="2026-09-07 13:03:42.440287355 +0000 UTC m=+639.471153701"
Sep 07 13:03:49.460114 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:49.460013    2565 generic.go:356] "Generic (PLEG): container finished" podID="08fad632-9c22-4303-8c09-aa25f954ccaa" containerID="e5b3805007d636cc1b4ef736498431e8cbf86f8a275de33c25a18e376e44561b" exitCode=0
Sep 07 13:03:49.460114 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:49.460093    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4" event={"ID":"08fad632-9c22-4303-8c09-aa25f954ccaa","Type":"ContainerDied","Data":"e5b3805007d636cc1b4ef736498431e8cbf86f8a275de33c25a18e376e44561b"}
Sep 07 13:03:49.460636 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:49.460529    2565 scope.go:117] "RemoveContainer" containerID="e5b3805007d636cc1b4ef736498431e8cbf86f8a275de33c25a18e376e44561b"
Sep 07 13:03:49.461531 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:49.461504    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:03:49.461606 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:49.461571    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:03:49.461644 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:49.461605    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:03:50.466153 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:50.466118    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4" event={"ID":"08fad632-9c22-4303-8c09-aa25f954ccaa","Type":"ContainerStarted","Data":"f59c907b38bca0f9f4dd42e22bb21f5f1c316c3d82d3a162d6c6bdd3bf23b4cf"}
Sep 07 13:03:50.813911 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:50.813870    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:03:50.814149 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:50.814126    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-pstv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Sep 07 13:03:50.814193 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:50.814177    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4" podUID="08fad632-9c22-4303-8c09-aa25f954ccaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Sep 07 13:03:51.814435 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:51.814392    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-pstv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Sep 07 13:03:51.814839 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:51.814457    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4" podUID="08fad632-9c22-4303-8c09-aa25f954ccaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Sep 07 13:03:52.813917 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:52.813872    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-pstv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Sep 07 13:03:52.814125 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:52.813937    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4" podUID="08fad632-9c22-4303-8c09-aa25f954ccaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Sep 07 13:03:53.423365 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:53.423316    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-gqz94"
Sep 07 13:03:53.814098 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:53.814059    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-pstv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Sep 07 13:03:53.814267 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:53.814119    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4" podUID="08fad632-9c22-4303-8c09-aa25f954ccaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Sep 07 13:03:54.813938 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:54.813891    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-pstv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body=
Sep 07 13:03:54.814360 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:54.813952    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4" podUID="08fad632-9c22-4303-8c09-aa25f954ccaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Sep 07 13:03:55.512322 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.512286    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"]
Sep 07 13:03:55.517562 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.517534    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.520123 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.520099    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 07 13:03:55.520975 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.520954    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 07 13:03:55.521081 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.520954    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 07 13:03:55.521081 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.521061    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4fn2j\""
Sep 07 13:03:55.533835 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.533804    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"]
Sep 07 13:03:55.538107 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.538083    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.538216 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.538132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.538274 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.538206    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.538274 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.538257    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.538398 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.538361    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.538446 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.538404    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddx2\" (UniqueName: \"kubernetes.io/projected/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-kube-api-access-nddx2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.639293 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.639239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.639507 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.639317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddx2\" (UniqueName: \"kubernetes.io/projected/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-kube-api-access-nddx2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.639507 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.639444    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.639624 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.639503    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.639624 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.639552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.639624 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.639579    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.639929 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.639899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.640235 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.639945    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.640235 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.640134    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.641983 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.641960    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.642194 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.642180    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.649186 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.649154    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddx2\" (UniqueName: \"kubernetes.io/projected/41fcd958-faa4-4ed0-8d12-8ee3c19f2d22-kube-api-access-nddx2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7\" (UID: \"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:55.818251 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.818220    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:03:55.818741 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.818673    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:03:55.819319 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.819299    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pstv4"
Sep 07 13:03:55.829060 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:55.829041    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:03:56.107558 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:56.107476    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"]
Sep 07 13:03:56.111449 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:03:56.111404    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fcd958_faa4_4ed0_8d12_8ee3c19f2d22.slice/crio-857472d68a5f8e7645465fd4abfb810f243db23e004fffef3b2122131904944f WatchSource:0}: Error finding container 857472d68a5f8e7645465fd4abfb810f243db23e004fffef3b2122131904944f: Status 404 returned error can't find the container with id 857472d68a5f8e7645465fd4abfb810f243db23e004fffef3b2122131904944f
Sep 07 13:03:56.133828 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:56.133799    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:03:56.501351 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:56.501233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7" event={"ID":"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22","Type":"ContainerStarted","Data":"857472d68a5f8e7645465fd4abfb810f243db23e004fffef3b2122131904944f"}
Sep 07 13:03:57.506702 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:57.506657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7" event={"ID":"41fcd958-faa4-4ed0-8d12-8ee3c19f2d22","Type":"ContainerStarted","Data":"9146b371a51a04abbaaed18ca07542abd3d04a9c9f8600117442df59e8d8203a"}
Sep 07 13:03:57.545917 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:03:57.545857    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7" podStartSLOduration=1.392135896 podStartE2EDuration="2.545840272s" totalImagesPullingTime="1.153704376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:03:57.542033442 +0000 UTC m=+654.572899752" watchObservedRunningTime="2026-09-07 13:03:57.545840272 +0000 UTC m=+654.576706582"
Sep 07 13:04:05.829781 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:05.829747    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:04:05.830287 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:05.829829    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:04:05.842382 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:05.842321    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:04:06.563923 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:06.563894    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7"
Sep 07 13:04:07.410938 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:07.410905    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:04:17.525108 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:17.525072    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:04:27.212299 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:27.212261    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:04:33.917560 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:33.917515    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:04:41.296078 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.296039    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"]
Sep 07 13:04:41.299951 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.299929    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.303840 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.303818    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 07 13:04:41.312758 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.312734    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"]
Sep 07 13:04:41.365875 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.365840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.366039 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.365884    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.366039 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.365905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.366039 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.365947    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.366039 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.365976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.366173 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.366035    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbjvm\" (UniqueName: \"kubernetes.io/projected/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-kube-api-access-jbjvm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.466896 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.466855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.467103 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.466916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.467103 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.466956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbjvm\" (UniqueName: \"kubernetes.io/projected/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-kube-api-access-jbjvm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.467103 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.467029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.467270 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.467094    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.467270 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.467136    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.467395 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.467353    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.467547 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.467518    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.467746 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.467725    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.469941 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.469921    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.470203 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.470168    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.481962 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.481930    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbjvm\" (UniqueName: \"kubernetes.io/projected/2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f-kube-api-access-jbjvm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdbxn\" (UID: \"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.610148 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.610047    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:41.899454 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:41.899342    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"]
Sep 07 13:04:41.901749 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:04:41.901712    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf15fcf_9d6d_4cba_b1d7_9e025c3ca11f.slice/crio-e2da39f1983d8a01c3d391c76e527273cdf8e10de8d099b55e3155254455603f WatchSource:0}: Error finding container e2da39f1983d8a01c3d391c76e527273cdf8e10de8d099b55e3155254455603f: Status 404 returned error can't find the container with id e2da39f1983d8a01c3d391c76e527273cdf8e10de8d099b55e3155254455603f
Sep 07 13:04:42.723180 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:42.723139    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn" event={"ID":"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f","Type":"ContainerStarted","Data":"26f02a511e78f148c4b68df10fa1619b19c8bd6f0609509ff33b6cc72c10beab"}
Sep 07 13:04:42.723588 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:42.723205    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn" event={"ID":"2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f","Type":"ContainerStarted","Data":"e2da39f1983d8a01c3d391c76e527273cdf8e10de8d099b55e3155254455603f"}
Sep 07 13:04:42.755942 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:42.755877    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn" podStartSLOduration=1.5727847910000001 podStartE2EDuration="1.755859281s" totalImagesPullingTime="183.07449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:04:42.75399636 +0000 UTC m=+699.784862675" watchObservedRunningTime="2026-09-07 13:04:42.755859281 +0000 UTC m=+699.786725620"
Sep 07 13:04:43.130578 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:43.130539    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:04:50.492090 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.492053    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"]
Sep 07 13:04:50.496037 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.496014    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.498275 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.498255    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 07 13:04:50.507384 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.507356    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"]
Sep 07 13:04:50.550247 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.550207    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1781c947-2aaf-43b7-9773-5daac436de70-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.550247 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.550261    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1781c947-2aaf-43b7-9773-5daac436de70-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.550490 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.550315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1781c947-2aaf-43b7-9773-5daac436de70-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.550490 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.550454    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1781c947-2aaf-43b7-9773-5daac436de70-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.550587 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.550497    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1781c947-2aaf-43b7-9773-5daac436de70-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.550587 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.550521    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p8d2\" (UniqueName: \"kubernetes.io/projected/1781c947-2aaf-43b7-9773-5daac436de70-kube-api-access-6p8d2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.651729 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.651689    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1781c947-2aaf-43b7-9773-5daac436de70-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.651899 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.651735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1781c947-2aaf-43b7-9773-5daac436de70-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.651899 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.651834    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1781c947-2aaf-43b7-9773-5daac436de70-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.652013 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.651933    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1781c947-2aaf-43b7-9773-5daac436de70-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.652013 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.651961    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1781c947-2aaf-43b7-9773-5daac436de70-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.652013 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.651988    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p8d2\" (UniqueName: \"kubernetes.io/projected/1781c947-2aaf-43b7-9773-5daac436de70-kube-api-access-6p8d2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.652255 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.652190    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1781c947-2aaf-43b7-9773-5daac436de70-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.652255 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.652229    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1781c947-2aaf-43b7-9773-5daac436de70-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.652433 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.652309    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1781c947-2aaf-43b7-9773-5daac436de70-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.654069 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.654042    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1781c947-2aaf-43b7-9773-5daac436de70-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.654440 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.654421    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1781c947-2aaf-43b7-9773-5daac436de70-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.661153 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.661129    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p8d2\" (UniqueName: \"kubernetes.io/projected/1781c947-2aaf-43b7-9773-5daac436de70-kube-api-access-6p8d2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fsqls\" (UID: \"1781c947-2aaf-43b7-9773-5daac436de70\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:50.807867 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:50.807825    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:04:51.084704 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:51.084677    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"]
Sep 07 13:04:51.087030 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:04:51.087004    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1781c947_2aaf_43b7_9773_5daac436de70.slice/crio-d467cfcd42de3ea2cfaa1b0ac12c9fc322c74da217fe4a6e27ab0ecd373dedc2 WatchSource:0}: Error finding container d467cfcd42de3ea2cfaa1b0ac12c9fc322c74da217fe4a6e27ab0ecd373dedc2: Status 404 returned error can't find the container with id d467cfcd42de3ea2cfaa1b0ac12c9fc322c74da217fe4a6e27ab0ecd373dedc2
Sep 07 13:04:51.610401 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:51.610321    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:51.610837 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:51.610421    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:51.623187 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:51.623157    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:51.768749 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:51.768714    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls" event={"ID":"1781c947-2aaf-43b7-9773-5daac436de70","Type":"ContainerStarted","Data":"0ac1769e1b0abcb2a8afce03cf5d471e501776d6cedb60663de6421bd4a5418c"}
Sep 07 13:04:51.768954 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:51.768759    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls" event={"ID":"1781c947-2aaf-43b7-9773-5daac436de70","Type":"ContainerStarted","Data":"d467cfcd42de3ea2cfaa1b0ac12c9fc322c74da217fe4a6e27ab0ecd373dedc2"}
Sep 07 13:04:51.780206 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:51.780171    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdbxn"
Sep 07 13:04:51.789043 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:51.788989    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls" podStartSLOduration=1.6244890189999999 podStartE2EDuration="1.78896934s" totalImagesPullingTime="164.480321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:04:51.788562382 +0000 UTC m=+708.819428693" watchObservedRunningTime="2026-09-07 13:04:51.78896934 +0000 UTC m=+708.819835651"
Sep 07 13:04:52.706130 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:04:52.706098    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:05:00.808774 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:05:00.808658    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:05:00.808774 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:05:00.808721    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:05:00.828784 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:05:00.828756    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:05:01.830935 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:05:01.830903    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fsqls"
Sep 07 13:06:25.776008 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:25.775926    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"]
Sep 07 13:06:25.779429 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:25.779414    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:25.782271 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:25.782249    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7ce94d-openshift-default-dockercfg-gxj6m\""
Sep 07 13:06:25.790685 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:25.790660    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"]
Sep 07 13:06:25.958912 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:25.958875    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-workload-certs\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:25.959075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:25.958925    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e53887c-3654-486d-85a6-191e7dbea57a-istiod-ca-cert\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:25.959075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:25.958960    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e53887c-3654-486d-85a6-191e7dbea57a-istio-podinfo\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:25.959075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:25.959059    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-istio-envoy\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:25.959181 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:25.959105    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-credential-socket\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:25.959181 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:25.959133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw5fq\" (UniqueName: \"kubernetes.io/projected/4e53887c-3654-486d-85a6-191e7dbea57a-kube-api-access-hw5fq\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:25.959249 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:25.959181    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-workload-socket\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:25.959249 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:25.959206    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-istio-data\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:25.959249 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:25.959226    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e53887c-3654-486d-85a6-191e7dbea57a-istio-token\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.060288 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.060252    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-workload-certs\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.060447 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.060299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e53887c-3654-486d-85a6-191e7dbea57a-istiod-ca-cert\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.060447 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.060349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e53887c-3654-486d-85a6-191e7dbea57a-istio-podinfo\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.060447 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.060395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-istio-envoy\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.060447 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.060437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-credential-socket\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.060626 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.060596    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw5fq\" (UniqueName: \"kubernetes.io/projected/4e53887c-3654-486d-85a6-191e7dbea57a-kube-api-access-hw5fq\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.060729 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.060708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-workload-socket\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.060815 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.060750    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-workload-certs\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.060815 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.060751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-istio-data\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.060815 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.060797    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-credential-socket\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.060978 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.060815    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e53887c-3654-486d-85a6-191e7dbea57a-istio-token\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.061082 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.061057    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-istio-data\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.061082 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.061078    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-workload-socket\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.061463 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.061114    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e53887c-3654-486d-85a6-191e7dbea57a-istiod-ca-cert\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.062831 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.062808    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-istio-envoy\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.063100 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.063083    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e53887c-3654-486d-85a6-191e7dbea57a-istio-podinfo\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.070687 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.070666    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e53887c-3654-486d-85a6-191e7dbea57a-istio-token\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.070784 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.070746    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw5fq\" (UniqueName: \"kubernetes.io/projected/4e53887c-3654-486d-85a6-191e7dbea57a-kube-api-access-hw5fq\") pod \"e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") " pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.090762 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.090731    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:26.370950 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.370923    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"]
Sep 07 13:06:26.373359 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:06:26.373306    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e53887c_3654_486d_85a6_191e7dbea57a.slice/crio-2c1276f7052fc6b2ae02ebd24870cb9af32a7ec0873553af93b43871f0471142 WatchSource:0}: Error finding container 2c1276f7052fc6b2ae02ebd24870cb9af32a7ec0873553af93b43871f0471142: Status 404 returned error can't find the container with id 2c1276f7052fc6b2ae02ebd24870cb9af32a7ec0873553af93b43871f0471142
Sep 07 13:06:26.375601 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.375562    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:06:26.375682 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.375648    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:06:26.375733 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:26.375696    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:06:27.239502 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:27.239464    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc" event={"ID":"4e53887c-3654-486d-85a6-191e7dbea57a","Type":"ContainerStarted","Data":"608bb006486854f9704fc397e8f7832d748669fd60a7f773a4a179bdf8fe743e"}
Sep 07 13:06:27.239976 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:27.239522    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc" event={"ID":"4e53887c-3654-486d-85a6-191e7dbea57a","Type":"ContainerStarted","Data":"2c1276f7052fc6b2ae02ebd24870cb9af32a7ec0873553af93b43871f0471142"}
Sep 07 13:06:27.275492 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:27.275438    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc" podStartSLOduration=2.275420989 podStartE2EDuration="2.275420989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:06:27.272541559 +0000 UTC m=+804.303407881" watchObservedRunningTime="2026-09-07 13:06:27.275420989 +0000 UTC m=+804.306287351"
Sep 07 13:06:28.091490 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:28.091448    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:28.096219 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:28.096194    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:28.244671 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:28.244631    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:28.245603 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:28.245579    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:29.513971 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:29.513938    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:06:35.650511 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.650462    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"]
Sep 07 13:06:35.654309 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.654292    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.657216 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.657192    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-e9d55a-openshift-default-dockercfg-bgkdf\""
Sep 07 13:06:35.668105 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.668081    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"]
Sep 07 13:06:35.755741 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.755705    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-envoy\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.755741 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.755745    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-podinfo\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.755978 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.755770    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-credential-socket\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.755978 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.755839    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fr44\" (UniqueName: \"kubernetes.io/projected/d635af4b-a988-4a4f-9188-24ddcf89ce64-kube-api-access-7fr44\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.755978 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.755883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-workload-socket\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.756092 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.755976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-token\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.756092 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.756040    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-workload-certs\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.756092 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.756064    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d635af4b-a988-4a4f-9188-24ddcf89ce64-istiod-ca-cert\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.756207 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.756163    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-data\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.856876 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.856844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-data\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.857084 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.856882    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-envoy\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.857084 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.856904    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-podinfo\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.857084 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.856931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-credential-socket\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.857084 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.856960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fr44\" (UniqueName: \"kubernetes.io/projected/d635af4b-a988-4a4f-9188-24ddcf89ce64-kube-api-access-7fr44\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.857084 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.856997    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-workload-socket\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.857084 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.857051    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-token\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.857084 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.857083    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-workload-certs\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.857513 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.857110    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d635af4b-a988-4a4f-9188-24ddcf89ce64-istiod-ca-cert\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.857513 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.857310    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-data\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.857513 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.857415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-credential-socket\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.857657 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.857519    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-workload-socket\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.857716 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.857663    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-workload-certs\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.857929 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.857812    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d635af4b-a988-4a4f-9188-24ddcf89ce64-istiod-ca-cert\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.859496 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.859476    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-envoy\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.859628 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.859612    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-podinfo\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.868409 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.868378    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-token\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.868541 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.868509    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fr44\" (UniqueName: \"kubernetes.io/projected/d635af4b-a988-4a4f-9188-24ddcf89ce64-kube-api-access-7fr44\") pod \"e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") " pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:35.965395 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:35.965272    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:36.264666 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:36.264639    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"]
Sep 07 13:06:36.267414 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:06:36.267386    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd635af4b_a988_4a4f_9188_24ddcf89ce64.slice/crio-e702d5da7bec090f5fd6825ae0afdac0b6b4b1a8bc69a1ccfca5b297121d05c9 WatchSource:0}: Error finding container e702d5da7bec090f5fd6825ae0afdac0b6b4b1a8bc69a1ccfca5b297121d05c9: Status 404 returned error can't find the container with id e702d5da7bec090f5fd6825ae0afdac0b6b4b1a8bc69a1ccfca5b297121d05c9
Sep 07 13:06:36.269903 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:36.269875    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:06:36.269980 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:36.269941    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:06:36.269980 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:36.269971    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:06:36.289848 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:36.289826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww" event={"ID":"d635af4b-a988-4a4f-9188-24ddcf89ce64","Type":"ContainerStarted","Data":"e702d5da7bec090f5fd6825ae0afdac0b6b4b1a8bc69a1ccfca5b297121d05c9"}
Sep 07 13:06:37.297686 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:37.297650    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww" event={"ID":"d635af4b-a988-4a4f-9188-24ddcf89ce64","Type":"ContainerStarted","Data":"10872495be30c240652c098d272d460f242f89a3da5c27134a78ebb2233c5184"}
Sep 07 13:06:37.322153 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:37.322102    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww" podStartSLOduration=2.322085934 podStartE2EDuration="2.322085934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:06:37.319637105 +0000 UTC m=+814.350503419" watchObservedRunningTime="2026-09-07 13:06:37.322085934 +0000 UTC m=+814.352952245"
Sep 07 13:06:37.965736 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:37.965703    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:37.970277 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:37.970253    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:38.305471 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:38.305440    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:38.306529 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:38.306510    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:39.929168 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:39.929130    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"]
Sep 07 13:06:39.932666 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:39.932650    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"
Sep 07 13:06:39.935065 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:39.935047    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2cd995c1\""
Sep 07 13:06:39.945884 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:39.945861    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"]
Sep 07 13:06:40.021351 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.018384    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8336cd-6c0d-45e6-8cee-bcc079388a55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7\" (UID: \"0b8336cd-6c0d-45e6-8cee-bcc079388a55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"
Sep 07 13:06:40.021351 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.018493    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr7vt\" (UniqueName: \"kubernetes.io/projected/0b8336cd-6c0d-45e6-8cee-bcc079388a55-kube-api-access-kr7vt\") pod \"payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7\" (UID: \"0b8336cd-6c0d-45e6-8cee-bcc079388a55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"
Sep 07 13:06:40.074250 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.074218    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"]
Sep 07 13:06:40.077885 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.077864    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"
Sep 07 13:06:40.086775 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.086695    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2cd995c1-dockercfg-rgwdw\""
Sep 07 13:06:40.090256 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.090228    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"]
Sep 07 13:06:40.119260 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.119228    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqpt9\" (UniqueName: \"kubernetes.io/projected/af0c2f74-b14e-4486-b6f7-c4fe49442483-kube-api-access-rqpt9\") pod \"payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg\" (UID: \"af0c2f74-b14e-4486-b6f7-c4fe49442483\") " pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"
Sep 07 13:06:40.119424 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.119364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8336cd-6c0d-45e6-8cee-bcc079388a55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7\" (UID: \"0b8336cd-6c0d-45e6-8cee-bcc079388a55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"
Sep 07 13:06:40.119424 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.119396    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af0c2f74-b14e-4486-b6f7-c4fe49442483-plugins-config-volume\") pod \"payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg\" (UID: \"af0c2f74-b14e-4486-b6f7-c4fe49442483\") " pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"
Sep 07 13:06:40.119510 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.119431    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr7vt\" (UniqueName: \"kubernetes.io/projected/0b8336cd-6c0d-45e6-8cee-bcc079388a55-kube-api-access-kr7vt\") pod \"payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7\" (UID: \"0b8336cd-6c0d-45e6-8cee-bcc079388a55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"
Sep 07 13:06:40.120104 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.120083    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8336cd-6c0d-45e6-8cee-bcc079388a55-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7\" (UID: \"0b8336cd-6c0d-45e6-8cee-bcc079388a55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"
Sep 07 13:06:40.129176 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.129147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr7vt\" (UniqueName: \"kubernetes.io/projected/0b8336cd-6c0d-45e6-8cee-bcc079388a55-kube-api-access-kr7vt\") pod \"payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7\" (UID: \"0b8336cd-6c0d-45e6-8cee-bcc079388a55\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"
Sep 07 13:06:40.221047 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.220947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af0c2f74-b14e-4486-b6f7-c4fe49442483-plugins-config-volume\") pod \"payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg\" (UID: \"af0c2f74-b14e-4486-b6f7-c4fe49442483\") " pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"
Sep 07 13:06:40.221047 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.221018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqpt9\" (UniqueName: \"kubernetes.io/projected/af0c2f74-b14e-4486-b6f7-c4fe49442483-kube-api-access-rqpt9\") pod \"payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg\" (UID: \"af0c2f74-b14e-4486-b6f7-c4fe49442483\") " pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"
Sep 07 13:06:40.221779 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.221752    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af0c2f74-b14e-4486-b6f7-c4fe49442483-plugins-config-volume\") pod \"payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg\" (UID: \"af0c2f74-b14e-4486-b6f7-c4fe49442483\") " pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"
Sep 07 13:06:40.232000 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.231977    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqpt9\" (UniqueName: \"kubernetes.io/projected/af0c2f74-b14e-4486-b6f7-c4fe49442483-kube-api-access-rqpt9\") pod \"payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg\" (UID: \"af0c2f74-b14e-4486-b6f7-c4fe49442483\") " pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"
Sep 07 13:06:40.243781 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.243756    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"
Sep 07 13:06:40.404501 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.404453    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"
Sep 07 13:06:40.543978 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.543950    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"]
Sep 07 13:06:40.546723 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:06:40.546693    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8336cd_6c0d_45e6_8cee_bcc079388a55.slice/crio-d06bd41e979928bd0d9d886f5f2fe4aa617e0d3b1aaa7bb46de495a5bbb2e659 WatchSource:0}: Error finding container d06bd41e979928bd0d9d886f5f2fe4aa617e0d3b1aaa7bb46de495a5bbb2e659: Status 404 returned error can't find the container with id d06bd41e979928bd0d9d886f5f2fe4aa617e0d3b1aaa7bb46de495a5bbb2e659
Sep 07 13:06:40.683650 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:40.681592    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"]
Sep 07 13:06:40.684204 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:06:40.684164    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0c2f74_b14e_4486_b6f7_c4fe49442483.slice/crio-ccf3e12dcbdf6f32379c3c6adf1b7b173463d71984ec3866007ee63a5cb800fe WatchSource:0}: Error finding container ccf3e12dcbdf6f32379c3c6adf1b7b173463d71984ec3866007ee63a5cb800fe: Status 404 returned error can't find the container with id ccf3e12dcbdf6f32379c3c6adf1b7b173463d71984ec3866007ee63a5cb800fe
Sep 07 13:06:41.325476 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:41.325444    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg" event={"ID":"af0c2f74-b14e-4486-b6f7-c4fe49442483","Type":"ContainerStarted","Data":"e5ff8e64cdf71fbb03cc881ebf7257be33e51ca79beacde451c3a5458af58aad"}
Sep 07 13:06:41.325936 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:41.325918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg" event={"ID":"af0c2f74-b14e-4486-b6f7-c4fe49442483","Type":"ContainerStarted","Data":"ccf3e12dcbdf6f32379c3c6adf1b7b173463d71984ec3866007ee63a5cb800fe"}
Sep 07 13:06:41.326025 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:41.326010    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"
Sep 07 13:06:41.327241 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:41.327213    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7" event={"ID":"0b8336cd-6c0d-45e6-8cee-bcc079388a55","Type":"ContainerStarted","Data":"be6ee3fdb6e22cc74aafd52839b2d9066eddcb7f8adb78442d8fa58048c0f547"}
Sep 07 13:06:41.327390 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:41.327262    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7" event={"ID":"0b8336cd-6c0d-45e6-8cee-bcc079388a55","Type":"ContainerStarted","Data":"d06bd41e979928bd0d9d886f5f2fe4aa617e0d3b1aaa7bb46de495a5bbb2e659"}
Sep 07 13:06:41.327390 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:41.327352    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"
Sep 07 13:06:41.345882 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:41.345833    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg" podStartSLOduration=1.3458185550000001 podStartE2EDuration="1.345818555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:06:41.343985344 +0000 UTC m=+818.374851659" watchObservedRunningTime="2026-09-07 13:06:41.345818555 +0000 UTC m=+818.376684862"
Sep 07 13:06:41.362368 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:41.362298    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7" podStartSLOduration=2.362284119 podStartE2EDuration="2.362284119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:06:41.362066326 +0000 UTC m=+818.392932637" watchObservedRunningTime="2026-09-07 13:06:41.362284119 +0000 UTC m=+818.393150464"
Sep 07 13:06:45.206781 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:45.206746    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:06:46.796599 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.796555    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"]
Sep 07 13:06:46.802868 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.802833    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:46.805498 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.805469    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-c8f26638-kserve-self-signed-certs\""
Sep 07 13:06:46.814113 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.814081    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"]
Sep 07 13:06:46.892949 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.892896    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gr7w\" (UniqueName: \"kubernetes.io/projected/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-kube-api-access-6gr7w\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:46.893262 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.893235    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-home\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:46.893467 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.893448    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-tls-certs\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:46.893604 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.893586    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-model-cache\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:46.893732 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.893716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-dshm\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:46.893858 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.893842    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-tmp-dir\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:46.996494 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.995039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-tls-certs\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:46.996494 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.995102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-model-cache\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:46.996494 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.995138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-dshm\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:46.996494 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.995183    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-tmp-dir\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:46.996494 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.995269    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gr7w\" (UniqueName: \"kubernetes.io/projected/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-kube-api-access-6gr7w\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:46.996494 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.995356    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-home\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:46.996494 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.995809    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-home\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:47.000895 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.996693    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-tmp-dir\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:47.000895 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:46.996791    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-model-cache\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:47.007347 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:47.006961    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-dshm\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:47.012372 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:47.012301    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-tls-certs\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:47.020062 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:47.020033    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gr7w\" (UniqueName: \"kubernetes.io/projected/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-kube-api-access-6gr7w\") pod \"e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") " pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:47.116943 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:47.116910    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:47.361580 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:47.361545    2565 generic.go:356] "Generic (PLEG): container finished" podID="4e53887c-3654-486d-85a6-191e7dbea57a" containerID="608bb006486854f9704fc397e8f7832d748669fd60a7f773a4a179bdf8fe743e" exitCode=0
Sep 07 13:06:47.361770 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:47.361623    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc" event={"ID":"4e53887c-3654-486d-85a6-191e7dbea57a","Type":"ContainerDied","Data":"608bb006486854f9704fc397e8f7832d748669fd60a7f773a4a179bdf8fe743e"}
Sep 07 13:06:47.362111 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:47.362094    2565 scope.go:117] "RemoveContainer" containerID="608bb006486854f9704fc397e8f7832d748669fd60a7f773a4a179bdf8fe743e"
Sep 07 13:06:47.363221 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:47.363193    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:06:47.363299 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:47.363263    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:06:47.363361 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:47.363297    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:06:47.427427 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:47.427078    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"]
Sep 07 13:06:48.368811 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:48.368772    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r" event={"ID":"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d","Type":"ContainerStarted","Data":"2e92b6b4a104912fbfce289e63e4056f4276ce12bd842511eb783c275c1fcda2"}
Sep 07 13:06:48.369290 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:48.368822    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r" event={"ID":"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d","Type":"ContainerStarted","Data":"4c69d8633563b385f0822d0e611c5d00dfa9701bab75b8c17826324cc8f5d82b"}
Sep 07 13:06:48.370759 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:48.370735    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc" event={"ID":"4e53887c-3654-486d-85a6-191e7dbea57a","Type":"ContainerStarted","Data":"733e3e90f13b3d79ba6128865edeba6b79c56f859640e8e90c8f29517e06887b"}
Sep 07 13:06:48.389957 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:48.389893    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r" podStartSLOduration=2.389870099 podStartE2EDuration="2.389870099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:06:48.388994971 +0000 UTC m=+825.419861284" watchObservedRunningTime="2026-09-07 13:06:48.389870099 +0000 UTC m=+825.420736410"
Sep 07 13:06:49.091894 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:49.091849    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:49.092106 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:49.091934    2565 patch_prober.go:28] interesting pod/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 07 13:06:49.092106 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:49.092019    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc" podUID="4e53887c-3654-486d-85a6-191e7dbea57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 07 13:06:50.091399 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:50.091313    2565 patch_prober.go:28] interesting pod/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 07 13:06:50.091936 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:50.091436    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc" podUID="4e53887c-3654-486d-85a6-191e7dbea57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 07 13:06:51.091699 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:51.091650    2565 patch_prober.go:28] interesting pod/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 07 13:06:51.092202 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:51.091739    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc" podUID="4e53887c-3654-486d-85a6-191e7dbea57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 07 13:06:52.092269 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:52.091823    2565 patch_prober.go:28] interesting pod/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 07 13:06:52.092269 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:52.091892    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc" podUID="4e53887c-3654-486d-85a6-191e7dbea57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 07 13:06:52.335282 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:52.335253    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"
Sep 07 13:06:53.092178 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:53.092128    2565 patch_prober.go:28] interesting pod/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 07 13:06:53.092178 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:53.092202    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc" podUID="4e53887c-3654-486d-85a6-191e7dbea57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 07 13:06:53.340511 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:53.340472    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"
Sep 07 13:06:54.092017 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.091966    2565 patch_prober.go:28] interesting pod/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 07 13:06:54.092215 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.092036    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc" podUID="4e53887c-3654-486d-85a6-191e7dbea57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 07 13:06:54.347553 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.347466    2565 patch_prober.go:28] interesting pod/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 13:06:54.347936 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.347561    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww" podUID="d635af4b-a988-4a4f-9188-24ddcf89ce64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 13:06:54.875028 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.874987    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"]
Sep 07 13:06:54.879772 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.879750    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:54.882810 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.882784    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-76f028-openshift-default-dockercfg-5rwgc\""
Sep 07 13:06:54.890967 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.890932    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"]
Sep 07 13:06:54.988634 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.988597    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-workload-socket\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:54.988634 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.988635    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ece672-58d2-489b-a652-9308ee1960d5-istio-token\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:54.988891 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.988683    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-credential-socket\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:54.988891 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.988728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-workload-certs\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:54.988891 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.988745    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-istio-envoy\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:54.988891 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.988758    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-istio-data\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:54.988891 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.988800    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk2nd\" (UniqueName: \"kubernetes.io/projected/30ece672-58d2-489b-a652-9308ee1960d5-kube-api-access-qk2nd\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:54.988891 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.988852    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ece672-58d2-489b-a652-9308ee1960d5-istiod-ca-cert\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:54.988891 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:54.988868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ece672-58d2-489b-a652-9308ee1960d5-istio-podinfo\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.089532 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.089485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-workload-socket\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.089532 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.089530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ece672-58d2-489b-a652-9308ee1960d5-istio-token\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.089792 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.089566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-credential-socket\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.089792 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.089593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-workload-certs\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.089792 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.089610    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-istio-envoy\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.089792 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.089631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-istio-data\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.089792 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.089657    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk2nd\" (UniqueName: \"kubernetes.io/projected/30ece672-58d2-489b-a652-9308ee1960d5-kube-api-access-qk2nd\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.089792 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.089708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ece672-58d2-489b-a652-9308ee1960d5-istiod-ca-cert\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.089792 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.089725    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ece672-58d2-489b-a652-9308ee1960d5-istio-podinfo\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.090572 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.089917    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-workload-socket\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.090572 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.090012    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-workload-certs\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.090572 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.090164    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-istio-data\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.090572 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.090392    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-credential-socket\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.090572 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.090435    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ece672-58d2-489b-a652-9308ee1960d5-istiod-ca-cert\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.092861 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.092833    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-istio-envoy\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.093137 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.093112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ece672-58d2-489b-a652-9308ee1960d5-istio-podinfo\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.095708 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.095683    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:55.096061 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.096041    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:55.096838 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.096815    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:06:55.099079 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.099060    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ece672-58d2-489b-a652-9308ee1960d5-istio-token\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.099412 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.099369    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk2nd\" (UniqueName: \"kubernetes.io/projected/30ece672-58d2-489b-a652-9308ee1960d5-kube-api-access-qk2nd\") pod \"e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") " pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.191898 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.191816    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:55.432151 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.432122    2565 generic.go:356] "Generic (PLEG): container finished" podID="d635af4b-a988-4a4f-9188-24ddcf89ce64" containerID="10872495be30c240652c098d272d460f242f89a3da5c27134a78ebb2233c5184" exitCode=0
Sep 07 13:06:55.432657 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.432196    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww" event={"ID":"d635af4b-a988-4a4f-9188-24ddcf89ce64","Type":"ContainerDied","Data":"10872495be30c240652c098d272d460f242f89a3da5c27134a78ebb2233c5184"}
Sep 07 13:06:55.432746 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.432731    2565 scope.go:117] "RemoveContainer" containerID="10872495be30c240652c098d272d460f242f89a3da5c27134a78ebb2233c5184"
Sep 07 13:06:55.433774 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.433752    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:06:55.433844 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.433828    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:06:55.433899 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.433884    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:06:55.491340 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.491295    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"]
Sep 07 13:06:55.492783 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:06:55.492753    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ece672_58d2_489b_a652_9308ee1960d5.slice/crio-0f3be5cc9c4a763db05f3b84561ff84e31c05937a8b2973026e42a311e6d838e WatchSource:0}: Error finding container 0f3be5cc9c4a763db05f3b84561ff84e31c05937a8b2973026e42a311e6d838e: Status 404 returned error can't find the container with id 0f3be5cc9c4a763db05f3b84561ff84e31c05937a8b2973026e42a311e6d838e
Sep 07 13:06:55.495814 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.495774    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:06:55.495892 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.495868    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:06:55.495931 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:55.495921    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:06:56.443575 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:56.443528    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww" event={"ID":"d635af4b-a988-4a4f-9188-24ddcf89ce64","Type":"ContainerStarted","Data":"3b089c887cef180721eabb2fd79a9af5856b8e1a1eb4b763f0a8d7a4d916137d"}
Sep 07 13:06:56.445818 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:56.445771    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29" event={"ID":"30ece672-58d2-489b-a652-9308ee1960d5","Type":"ContainerStarted","Data":"ff51de4b535c3da7dfabf0f2947853c3734588b06387e4423bd619fc2387dab3"}
Sep 07 13:06:56.445968 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:56.445911    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29" event={"ID":"30ece672-58d2-489b-a652-9308ee1960d5","Type":"ContainerStarted","Data":"0f3be5cc9c4a763db05f3b84561ff84e31c05937a8b2973026e42a311e6d838e"}
Sep 07 13:06:56.490461 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:56.490392    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29" podStartSLOduration=2.490370644 podStartE2EDuration="2.490370644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:06:56.488818735 +0000 UTC m=+833.519685074" watchObservedRunningTime="2026-09-07 13:06:56.490370644 +0000 UTC m=+833.521236955"
Sep 07 13:06:56.965873 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:56.965833    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:06:56.966137 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:56.966050    2565 patch_prober.go:28] interesting pod/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 07 13:06:56.966137 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:56.966093    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww" podUID="d635af4b-a988-4a4f-9188-24ddcf89ce64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 07 13:06:57.117753 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:57.117714    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:57.117919 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:57.117770    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:57.129998 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:57.129970    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:57.194527 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:57.194496    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:57.202658 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:57.202628    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:57.452087 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:57.452048    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:57.453223 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:57.453200    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:06:57.463258 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:57.463238    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:06:57.966249 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:57.966213    2565 patch_prober.go:28] interesting pod/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 07 13:06:57.966462 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:57.966300    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww" podUID="d635af4b-a988-4a4f-9188-24ddcf89ce64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 07 13:06:58.965979 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:58.965942    2565 patch_prober.go:28] interesting pod/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 07 13:06:58.966482 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:58.966020    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww" podUID="d635af4b-a988-4a4f-9188-24ddcf89ce64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 07 13:06:59.965889 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:59.965854    2565 patch_prober.go:28] interesting pod/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 07 13:06:59.966076 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:06:59.965913    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww" podUID="d635af4b-a988-4a4f-9188-24ddcf89ce64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 07 13:07:00.966300 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:00.966267    2565 patch_prober.go:28] interesting pod/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 07 13:07:00.966763 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:00.966322    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww" podUID="d635af4b-a988-4a4f-9188-24ddcf89ce64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 07 13:07:01.970030 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:01.970002    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:07:01.970493 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:01.970294    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:07:01.971057 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:01.971038    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:07:03.165968 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:03.165932    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9"]
Sep 07 13:07:03.169817 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:03.169799    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9"
Sep 07 13:07:03.172498 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:03.172477    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 13:07:03.172690 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:03.172670    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flngs\""
Sep 07 13:07:03.187526 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:03.187499    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9"]
Sep 07 13:07:03.282775 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:03.282743    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdd745d-09cd-4da4-b75d-8e11c26c0a0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9\" (UID: \"2fdd745d-09cd-4da4-b75d-8e11c26c0a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9"
Sep 07 13:07:03.383547 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:03.383516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdd745d-09cd-4da4-b75d-8e11c26c0a0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9\" (UID: \"2fdd745d-09cd-4da4-b75d-8e11c26c0a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9"
Sep 07 13:07:03.386059 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:03.386033    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 13:07:03.394965 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:03.394938    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdd745d-09cd-4da4-b75d-8e11c26c0a0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9\" (UID: \"2fdd745d-09cd-4da4-b75d-8e11c26c0a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9"
Sep 07 13:07:03.483229 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:03.483147    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flngs\""
Sep 07 13:07:03.490823 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:03.490797    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9"
Sep 07 13:07:03.765438 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:03.765408    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9"]
Sep 07 13:07:03.767490 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:07:03.767460    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fdd745d_09cd_4da4_b75d_8e11c26c0a0d.slice/crio-09635f7a22143c45b1d38df650a7d818212d67a2bdd35ec3615fa680ff5343da WatchSource:0}: Error finding container 09635f7a22143c45b1d38df650a7d818212d67a2bdd35ec3615fa680ff5343da: Status 404 returned error can't find the container with id 09635f7a22143c45b1d38df650a7d818212d67a2bdd35ec3615fa680ff5343da
Sep 07 13:07:04.488505 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:04.488468    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9" event={"ID":"2fdd745d-09cd-4da4-b75d-8e11c26c0a0d","Type":"ContainerStarted","Data":"09635f7a22143c45b1d38df650a7d818212d67a2bdd35ec3615fa680ff5343da"}
Sep 07 13:07:04.616847 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:04.616811    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:07:07.514935 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:07.514837    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9" event={"ID":"2fdd745d-09cd-4da4-b75d-8e11c26c0a0d","Type":"ContainerStarted","Data":"e5e60bf37322cebb990d04000de24e6f8b9d44c2eb59c2c141cdf2ef2cf47b97"}
Sep 07 13:07:07.538547 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:07.538498    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9" podStartSLOduration=1.439617949 podStartE2EDuration="4.538483839s" totalImagesPullingTime="3.09886589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:07:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:07:07.538163848 +0000 UTC m=+844.569030157" watchObservedRunningTime="2026-09-07 13:07:07.538483839 +0000 UTC m=+844.569350156"
Sep 07 13:07:08.522054 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:08.522013    2565 generic.go:356] "Generic (PLEG): container finished" podID="2fdd745d-09cd-4da4-b75d-8e11c26c0a0d" containerID="e5e60bf37322cebb990d04000de24e6f8b9d44c2eb59c2c141cdf2ef2cf47b97" exitCode=0
Sep 07 13:07:08.522458 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:08.522097    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9" event={"ID":"2fdd745d-09cd-4da4-b75d-8e11c26c0a0d","Type":"ContainerDied","Data":"e5e60bf37322cebb990d04000de24e6f8b9d44c2eb59c2c141cdf2ef2cf47b97"}
Sep 07 13:07:09.709076 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:09.709050    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9"
Sep 07 13:07:09.756699 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:09.756659    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdd745d-09cd-4da4-b75d-8e11c26c0a0d-maas-api-service-ca\") pod \"2fdd745d-09cd-4da4-b75d-8e11c26c0a0d\" (UID: \"2fdd745d-09cd-4da4-b75d-8e11c26c0a0d\") "
Sep 07 13:07:09.757032 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:09.757010    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fdd745d-09cd-4da4-b75d-8e11c26c0a0d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2fdd745d-09cd-4da4-b75d-8e11c26c0a0d" (UID: "2fdd745d-09cd-4da4-b75d-8e11c26c0a0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:07:09.757155 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:09.757138    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdd745d-09cd-4da4-b75d-8e11c26c0a0d-maas-api-service-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:07:10.538004 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:10.537974    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9"
Sep 07 13:07:10.538190 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:10.538014    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9" event={"ID":"2fdd745d-09cd-4da4-b75d-8e11c26c0a0d","Type":"ContainerDied","Data":"09635f7a22143c45b1d38df650a7d818212d67a2bdd35ec3615fa680ff5343da"}
Sep 07 13:07:10.538190 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:10.538061    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09635f7a22143c45b1d38df650a7d818212d67a2bdd35ec3615fa680ff5343da"
Sep 07 13:07:13.495489 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:13.495453    2565 patch_prober.go:28] interesting pod/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 13:07:13.495895 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:13.495518    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29" podUID="30ece672-58d2-489b-a652-9308ee1960d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 13:07:15.568519 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:15.568483    2565 generic.go:356] "Generic (PLEG): container finished" podID="30ece672-58d2-489b-a652-9308ee1960d5" containerID="ff51de4b535c3da7dfabf0f2947853c3734588b06387e4423bd619fc2387dab3" exitCode=0
Sep 07 13:07:15.568973 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:15.568551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29" event={"ID":"30ece672-58d2-489b-a652-9308ee1960d5","Type":"ContainerDied","Data":"ff51de4b535c3da7dfabf0f2947853c3734588b06387e4423bd619fc2387dab3"}
Sep 07 13:07:15.569049 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:15.568978    2565 scope.go:117] "RemoveContainer" containerID="ff51de4b535c3da7dfabf0f2947853c3734588b06387e4423bd619fc2387dab3"
Sep 07 13:07:15.570018 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:15.569991    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:07:15.570097 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:15.570057    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:07:15.570097 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:15.570088    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:07:16.576058 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:16.576026    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29" event={"ID":"30ece672-58d2-489b-a652-9308ee1960d5","Type":"ContainerStarted","Data":"e2fb0187527a4bf473d2d3db76ab857a79953ee25b6964538497a8448d487f2c"}
Sep 07 13:07:17.192363 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:17.192301    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:07:17.192565 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:17.192489    2565 patch_prober.go:28] interesting pod/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 13:07:17.192565 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:17.192550    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29" podUID="30ece672-58d2-489b-a652-9308ee1960d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 13:07:18.193029 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:18.192521    2565 patch_prober.go:28] interesting pod/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 13:07:18.193029 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:18.192632    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29" podUID="30ece672-58d2-489b-a652-9308ee1960d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 13:07:19.192882 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:19.192843    2565 patch_prober.go:28] interesting pod/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 13:07:19.193231 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:19.193182    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29" podUID="30ece672-58d2-489b-a652-9308ee1960d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 13:07:19.741666 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:19.741625    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"]
Sep 07 13:07:19.741937 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:19.741910    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg" podUID="af0c2f74-b14e-4486-b6f7-c4fe49442483" containerName="payload-processing" containerID="cri-o://e5ff8e64cdf71fbb03cc881ebf7257be33e51ca79beacde451c3a5458af58aad" gracePeriod=30
Sep 07 13:07:19.748097 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:19.747987    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"]
Sep 07 13:07:19.748269 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:19.748185    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7" podUID="0b8336cd-6c0d-45e6-8cee-bcc079388a55" containerName="payload-pre-processing" containerID="cri-o://be6ee3fdb6e22cc74aafd52839b2d9066eddcb7f8adb78442d8fa58048c0f547" gracePeriod=30
Sep 07 13:07:19.938968 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:19.938935    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9"]
Sep 07 13:07:19.942164 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:19.942136    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cd995c1-hs6p9"]
Sep 07 13:07:20.122082 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.121733    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"
Sep 07 13:07:20.124253 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.124237    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"
Sep 07 13:07:20.206354 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.206108    2565 patch_prober.go:28] interesting pod/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 13:07:20.206354 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.206261    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29" podUID="30ece672-58d2-489b-a652-9308ee1960d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 13:07:20.275636 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.275599    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8336cd-6c0d-45e6-8cee-bcc079388a55-plugins-config-volume\") pod \"0b8336cd-6c0d-45e6-8cee-bcc079388a55\" (UID: \"0b8336cd-6c0d-45e6-8cee-bcc079388a55\") "
Sep 07 13:07:20.275801 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.275651    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqpt9\" (UniqueName: \"kubernetes.io/projected/af0c2f74-b14e-4486-b6f7-c4fe49442483-kube-api-access-rqpt9\") pod \"af0c2f74-b14e-4486-b6f7-c4fe49442483\" (UID: \"af0c2f74-b14e-4486-b6f7-c4fe49442483\") "
Sep 07 13:07:20.275801 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.275683    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr7vt\" (UniqueName: \"kubernetes.io/projected/0b8336cd-6c0d-45e6-8cee-bcc079388a55-kube-api-access-kr7vt\") pod \"0b8336cd-6c0d-45e6-8cee-bcc079388a55\" (UID: \"0b8336cd-6c0d-45e6-8cee-bcc079388a55\") "
Sep 07 13:07:20.275801 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.275716    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af0c2f74-b14e-4486-b6f7-c4fe49442483-plugins-config-volume\") pod \"af0c2f74-b14e-4486-b6f7-c4fe49442483\" (UID: \"af0c2f74-b14e-4486-b6f7-c4fe49442483\") "
Sep 07 13:07:20.276048 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.276009    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b8336cd-6c0d-45e6-8cee-bcc079388a55-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b8336cd-6c0d-45e6-8cee-bcc079388a55" (UID: "0b8336cd-6c0d-45e6-8cee-bcc079388a55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:07:20.276188 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.276095    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8336cd-6c0d-45e6-8cee-bcc079388a55-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:07:20.276188 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.276151    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0c2f74-b14e-4486-b6f7-c4fe49442483-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af0c2f74-b14e-4486-b6f7-c4fe49442483" (UID: "af0c2f74-b14e-4486-b6f7-c4fe49442483"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:07:20.278116 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.278094    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8336cd-6c0d-45e6-8cee-bcc079388a55-kube-api-access-kr7vt" (OuterVolumeSpecName: "kube-api-access-kr7vt") pod "0b8336cd-6c0d-45e6-8cee-bcc079388a55" (UID: "0b8336cd-6c0d-45e6-8cee-bcc079388a55"). InnerVolumeSpecName "kube-api-access-kr7vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:07:20.278185 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.278092    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0c2f74-b14e-4486-b6f7-c4fe49442483-kube-api-access-rqpt9" (OuterVolumeSpecName: "kube-api-access-rqpt9") pod "af0c2f74-b14e-4486-b6f7-c4fe49442483" (UID: "af0c2f74-b14e-4486-b6f7-c4fe49442483"). InnerVolumeSpecName "kube-api-access-rqpt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:07:20.377279 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.377188    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqpt9\" (UniqueName: \"kubernetes.io/projected/af0c2f74-b14e-4486-b6f7-c4fe49442483-kube-api-access-rqpt9\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:07:20.377279 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.377220    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr7vt\" (UniqueName: \"kubernetes.io/projected/0b8336cd-6c0d-45e6-8cee-bcc079388a55-kube-api-access-kr7vt\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:07:20.377279 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.377230    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af0c2f74-b14e-4486-b6f7-c4fe49442483-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:07:20.609572 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.609537    2565 generic.go:356] "Generic (PLEG): container finished" podID="af0c2f74-b14e-4486-b6f7-c4fe49442483" containerID="e5ff8e64cdf71fbb03cc881ebf7257be33e51ca79beacde451c3a5458af58aad" exitCode=0
Sep 07 13:07:20.609747 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.609588    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"
Sep 07 13:07:20.609747 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.609629    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg" event={"ID":"af0c2f74-b14e-4486-b6f7-c4fe49442483","Type":"ContainerDied","Data":"e5ff8e64cdf71fbb03cc881ebf7257be33e51ca79beacde451c3a5458af58aad"}
Sep 07 13:07:20.609747 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.609693    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg" event={"ID":"af0c2f74-b14e-4486-b6f7-c4fe49442483","Type":"ContainerDied","Data":"ccf3e12dcbdf6f32379c3c6adf1b7b173463d71984ec3866007ee63a5cb800fe"}
Sep 07 13:07:20.609747 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.609723    2565 scope.go:117] "RemoveContainer" containerID="e5ff8e64cdf71fbb03cc881ebf7257be33e51ca79beacde451c3a5458af58aad"
Sep 07 13:07:20.611436 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.611410    2565 generic.go:356] "Generic (PLEG): container finished" podID="0b8336cd-6c0d-45e6-8cee-bcc079388a55" containerID="be6ee3fdb6e22cc74aafd52839b2d9066eddcb7f8adb78442d8fa58048c0f547" exitCode=0
Sep 07 13:07:20.611556 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.611444    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"
Sep 07 13:07:20.611556 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.611474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7" event={"ID":"0b8336cd-6c0d-45e6-8cee-bcc079388a55","Type":"ContainerDied","Data":"be6ee3fdb6e22cc74aafd52839b2d9066eddcb7f8adb78442d8fa58048c0f547"}
Sep 07 13:07:20.611556 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.611520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7" event={"ID":"0b8336cd-6c0d-45e6-8cee-bcc079388a55","Type":"ContainerDied","Data":"d06bd41e979928bd0d9d886f5f2fe4aa617e0d3b1aaa7bb46de495a5bbb2e659"}
Sep 07 13:07:20.619733 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.619709    2565 scope.go:117] "RemoveContainer" containerID="e5ff8e64cdf71fbb03cc881ebf7257be33e51ca79beacde451c3a5458af58aad"
Sep 07 13:07:20.620413 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:07:20.620106    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ff8e64cdf71fbb03cc881ebf7257be33e51ca79beacde451c3a5458af58aad\": container with ID starting with e5ff8e64cdf71fbb03cc881ebf7257be33e51ca79beacde451c3a5458af58aad not found: ID does not exist" containerID="e5ff8e64cdf71fbb03cc881ebf7257be33e51ca79beacde451c3a5458af58aad"
Sep 07 13:07:20.620413 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.620150    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ff8e64cdf71fbb03cc881ebf7257be33e51ca79beacde451c3a5458af58aad"} err="failed to get container status \"e5ff8e64cdf71fbb03cc881ebf7257be33e51ca79beacde451c3a5458af58aad\": rpc error: code = NotFound desc = could not find container \"e5ff8e64cdf71fbb03cc881ebf7257be33e51ca79beacde451c3a5458af58aad\": container with ID starting with e5ff8e64cdf71fbb03cc881ebf7257be33e51ca79beacde451c3a5458af58aad not found: ID does not exist"
Sep 07 13:07:20.620413 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.620176    2565 scope.go:117] "RemoveContainer" containerID="be6ee3fdb6e22cc74aafd52839b2d9066eddcb7f8adb78442d8fa58048c0f547"
Sep 07 13:07:20.629871 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.629833    2565 scope.go:117] "RemoveContainer" containerID="be6ee3fdb6e22cc74aafd52839b2d9066eddcb7f8adb78442d8fa58048c0f547"
Sep 07 13:07:20.630170 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:07:20.630152    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6ee3fdb6e22cc74aafd52839b2d9066eddcb7f8adb78442d8fa58048c0f547\": container with ID starting with be6ee3fdb6e22cc74aafd52839b2d9066eddcb7f8adb78442d8fa58048c0f547 not found: ID does not exist" containerID="be6ee3fdb6e22cc74aafd52839b2d9066eddcb7f8adb78442d8fa58048c0f547"
Sep 07 13:07:20.630257 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.630177    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6ee3fdb6e22cc74aafd52839b2d9066eddcb7f8adb78442d8fa58048c0f547"} err="failed to get container status \"be6ee3fdb6e22cc74aafd52839b2d9066eddcb7f8adb78442d8fa58048c0f547\": rpc error: code = NotFound desc = could not find container \"be6ee3fdb6e22cc74aafd52839b2d9066eddcb7f8adb78442d8fa58048c0f547\": container with ID starting with be6ee3fdb6e22cc74aafd52839b2d9066eddcb7f8adb78442d8fa58048c0f547 not found: ID does not exist"
Sep 07 13:07:20.647067 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.647042    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"]
Sep 07 13:07:20.654669 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.654641    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cd995c1-776c67947-5rqpg"]
Sep 07 13:07:20.672417 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.672391    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"]
Sep 07 13:07:20.702351 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:20.701484    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cd995c1-5c7ccccc5d-shmv7"]
Sep 07 13:07:21.174038 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.174004    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"]
Sep 07 13:07:21.174564 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.174548    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b8336cd-6c0d-45e6-8cee-bcc079388a55" containerName="payload-pre-processing"
Sep 07 13:07:21.174564 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.174566    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b8336cd-6c0d-45e6-8cee-bcc079388a55" containerName="payload-pre-processing"
Sep 07 13:07:21.174680 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.174578    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af0c2f74-b14e-4486-b6f7-c4fe49442483" containerName="payload-processing"
Sep 07 13:07:21.174680 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.174584    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="af0c2f74-b14e-4486-b6f7-c4fe49442483" containerName="payload-processing"
Sep 07 13:07:21.174680 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.174612    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fdd745d-09cd-4da4-b75d-8e11c26c0a0d" containerName="revoke-keys"
Sep 07 13:07:21.174680 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.174618    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fdd745d-09cd-4da4-b75d-8e11c26c0a0d" containerName="revoke-keys"
Sep 07 13:07:21.174846 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.174683    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="af0c2f74-b14e-4486-b6f7-c4fe49442483" containerName="payload-processing"
Sep 07 13:07:21.174846 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.174695    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b8336cd-6c0d-45e6-8cee-bcc079388a55" containerName="payload-pre-processing"
Sep 07 13:07:21.174846 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.174704    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fdd745d-09cd-4da4-b75d-8e11c26c0a0d" containerName="revoke-keys"
Sep 07 13:07:21.186350 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.185590    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.195352 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.192255    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"]
Sep 07 13:07:21.195352 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.192514    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c8f5c2a5-gw-openshift-default-dockercfg-hw5sw\""
Sep 07 13:07:21.195584 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.195432    2565 patch_prober.go:28] interesting pod/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 13:07:21.195584 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.195493    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29" podUID="30ece672-58d2-489b-a652-9308ee1960d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 13:07:21.301348 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.298976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0303ad5e-d760-49de-ac87-0719578db8d6-istio-token\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.301348 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.299082    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-credential-socket\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.301348 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.299127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-workload-socket\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.301348 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.299255    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-istio-envoy\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.301348 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.299299    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0303ad5e-d760-49de-ac87-0719578db8d6-istiod-ca-cert\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.301348 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.299354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0303ad5e-d760-49de-ac87-0719578db8d6-istio-podinfo\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.301348 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.299382    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxgx\" (UniqueName: \"kubernetes.io/projected/0303ad5e-d760-49de-ac87-0719578db8d6-kube-api-access-bnxgx\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.301348 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.299434    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-istio-data\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.301348 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.299459    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-workload-certs\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.400137 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.400086    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-istio-data\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.400371 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.400148    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-workload-certs\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.400371 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.400191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0303ad5e-d760-49de-ac87-0719578db8d6-istio-token\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.400371 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.400255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-credential-socket\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.400371 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.400296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-workload-socket\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.400623 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.400395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-istio-envoy\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.400623 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.400437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0303ad5e-d760-49de-ac87-0719578db8d6-istiod-ca-cert\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.400623 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.400473    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0303ad5e-d760-49de-ac87-0719578db8d6-istio-podinfo\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.400623 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.400499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxgx\" (UniqueName: \"kubernetes.io/projected/0303ad5e-d760-49de-ac87-0719578db8d6-kube-api-access-bnxgx\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.401307 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.401274    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-istio-data\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.401617 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.401591    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-workload-certs\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.402147 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.402117    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-credential-socket\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.402425 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.402399    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-workload-socket\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.403405 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.403382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0303ad5e-d760-49de-ac87-0719578db8d6-istiod-ca-cert\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.405660 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.405637    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-istio-envoy\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.405750 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.405674    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0303ad5e-d760-49de-ac87-0719578db8d6-istio-podinfo\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.409543 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.409522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxgx\" (UniqueName: \"kubernetes.io/projected/0303ad5e-d760-49de-ac87-0719578db8d6-kube-api-access-bnxgx\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.409918 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.409898    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0303ad5e-d760-49de-ac87-0719578db8d6-istio-token\") pod \"e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") " pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.494076 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.493978    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b8336cd-6c0d-45e6-8cee-bcc079388a55" path="/var/lib/kubelet/pods/0b8336cd-6c0d-45e6-8cee-bcc079388a55/volumes"
Sep 07 13:07:21.494576 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.494550    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fdd745d-09cd-4da4-b75d-8e11c26c0a0d" path="/var/lib/kubelet/pods/2fdd745d-09cd-4da4-b75d-8e11c26c0a0d/volumes"
Sep 07 13:07:21.495074 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.495050    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af0c2f74-b14e-4486-b6f7-c4fe49442483" path="/var/lib/kubelet/pods/af0c2f74-b14e-4486-b6f7-c4fe49442483/volumes"
Sep 07 13:07:21.515785 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.515755    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:21.800929 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.800898    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"]
Sep 07 13:07:21.802965 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:07:21.802935    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0303ad5e_d760_49de_ac87_0719578db8d6.slice/crio-ec3c482c6b8e0577e69a8813e2a97c1185840e28a94d10e0b75a50ef5fd05e63 WatchSource:0}: Error finding container ec3c482c6b8e0577e69a8813e2a97c1185840e28a94d10e0b75a50ef5fd05e63: Status 404 returned error can't find the container with id ec3c482c6b8e0577e69a8813e2a97c1185840e28a94d10e0b75a50ef5fd05e63
Sep 07 13:07:21.805474 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.805434    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:07:21.805559 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.805526    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:07:21.805599 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:21.805568    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:07:22.196388 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:22.196289    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:07:22.196812 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:22.196800    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:07:22.197671 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:22.197651    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:07:22.628601 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:22.628558    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7" event={"ID":"0303ad5e-d760-49de-ac87-0719578db8d6","Type":"ContainerStarted","Data":"0d97f5b8833d86b79ab3f719694121b1cc7bda3eb6140bddbbb9641ec0686b79"}
Sep 07 13:07:22.629045 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:22.628622    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7" event={"ID":"0303ad5e-d760-49de-ac87-0719578db8d6","Type":"ContainerStarted","Data":"ec3c482c6b8e0577e69a8813e2a97c1185840e28a94d10e0b75a50ef5fd05e63"}
Sep 07 13:07:22.651751 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:22.651695    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7" podStartSLOduration=1.651679178 podStartE2EDuration="1.651679178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:07:22.649857177 +0000 UTC m=+859.680723486" watchObservedRunningTime="2026-09-07 13:07:22.651679178 +0000 UTC m=+859.682545487"
Sep 07 13:07:23.515921 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:23.515880    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:23.521699 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:23.521672    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:23.634296 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:23.634256    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:23.635407 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:23.635388    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:24.609344 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:24.609282    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:07:25.139338 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:25.139289    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"]
Sep 07 13:07:25.145904 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:25.145882    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"
Sep 07 13:07:25.148387 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:25.148365    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c8f5c2a5\""
Sep 07 13:07:25.157014 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:25.156987    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"]
Sep 07 13:07:25.243707 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:25.243667    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/429e1444-bd0a-4cb3-8efb-596bbdbc8d28-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt\" (UID: \"429e1444-bd0a-4cb3-8efb-596bbdbc8d28\") " pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"
Sep 07 13:07:25.243934 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:25.243739    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z9nn\" (UniqueName: \"kubernetes.io/projected/429e1444-bd0a-4cb3-8efb-596bbdbc8d28-kube-api-access-9z9nn\") pod \"payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt\" (UID: \"429e1444-bd0a-4cb3-8efb-596bbdbc8d28\") " pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"
Sep 07 13:07:25.344525 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:25.344485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z9nn\" (UniqueName: \"kubernetes.io/projected/429e1444-bd0a-4cb3-8efb-596bbdbc8d28-kube-api-access-9z9nn\") pod \"payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt\" (UID: \"429e1444-bd0a-4cb3-8efb-596bbdbc8d28\") " pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"
Sep 07 13:07:25.344719 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:25.344616    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/429e1444-bd0a-4cb3-8efb-596bbdbc8d28-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt\" (UID: \"429e1444-bd0a-4cb3-8efb-596bbdbc8d28\") " pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"
Sep 07 13:07:25.345254 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:25.345233    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/429e1444-bd0a-4cb3-8efb-596bbdbc8d28-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt\" (UID: \"429e1444-bd0a-4cb3-8efb-596bbdbc8d28\") " pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"
Sep 07 13:07:25.355227 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:25.355193    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z9nn\" (UniqueName: \"kubernetes.io/projected/429e1444-bd0a-4cb3-8efb-596bbdbc8d28-kube-api-access-9z9nn\") pod \"payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt\" (UID: \"429e1444-bd0a-4cb3-8efb-596bbdbc8d28\") " pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"
Sep 07 13:07:25.457922 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:25.457591    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"
Sep 07 13:07:25.757714 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:25.757689    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"]
Sep 07 13:07:25.759883 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:07:25.759838    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429e1444_bd0a_4cb3_8efb_596bbdbc8d28.slice/crio-50bc8251c39760cd24efe561537d64b8b258e02610c989b0d5bf42f1ceee8944 WatchSource:0}: Error finding container 50bc8251c39760cd24efe561537d64b8b258e02610c989b0d5bf42f1ceee8944: Status 404 returned error can't find the container with id 50bc8251c39760cd24efe561537d64b8b258e02610c989b0d5bf42f1ceee8944
Sep 07 13:07:26.658015 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:26.657963    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt" event={"ID":"429e1444-bd0a-4cb3-8efb-596bbdbc8d28","Type":"ContainerStarted","Data":"6da78eafbb87d0a4b880e62e5f62e8301a2ed2aa94166697d242d1877dfb0993"}
Sep 07 13:07:26.658529 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:26.658031    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt" event={"ID":"429e1444-bd0a-4cb3-8efb-596bbdbc8d28","Type":"ContainerStarted","Data":"50bc8251c39760cd24efe561537d64b8b258e02610c989b0d5bf42f1ceee8944"}
Sep 07 13:07:26.658529 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:26.658060    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"
Sep 07 13:07:26.675816 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:26.675761    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt" podStartSLOduration=1.67574172 podStartE2EDuration="1.67574172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:07:26.674887638 +0000 UTC m=+863.705753961" watchObservedRunningTime="2026-09-07 13:07:26.67574172 +0000 UTC m=+863.706608031"
Sep 07 13:07:37.666033 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:37.665998    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"
Sep 07 13:07:41.928501 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:41.928466    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:07:45.768311 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:45.768219    2565 generic.go:356] "Generic (PLEG): container finished" podID="0303ad5e-d760-49de-ac87-0719578db8d6" containerID="0d97f5b8833d86b79ab3f719694121b1cc7bda3eb6140bddbbb9641ec0686b79" exitCode=0
Sep 07 13:07:45.768708 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:45.768297    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7" event={"ID":"0303ad5e-d760-49de-ac87-0719578db8d6","Type":"ContainerDied","Data":"0d97f5b8833d86b79ab3f719694121b1cc7bda3eb6140bddbbb9641ec0686b79"}
Sep 07 13:07:45.768751 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:45.768708    2565 scope.go:117] "RemoveContainer" containerID="0d97f5b8833d86b79ab3f719694121b1cc7bda3eb6140bddbbb9641ec0686b79"
Sep 07 13:07:45.769768 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:45.769743    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:07:45.769823 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:45.769811    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:07:45.769863 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:45.769844    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:07:46.778592 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:46.778558    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7" event={"ID":"0303ad5e-d760-49de-ac87-0719578db8d6","Type":"ContainerStarted","Data":"9fefa3739b15252931419d8f10dea9047b873cfa8a2e67788ce984d28bd7fd9d"}
Sep 07 13:07:47.516182 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:47.516143    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:47.516437 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:47.516414    2565 patch_prober.go:28] interesting pod/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 07 13:07:47.516534 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:47.516474    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7" podUID="0303ad5e-d760-49de-ac87-0719578db8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 07 13:07:48.516887 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:48.516847    2565 patch_prober.go:28] interesting pod/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 07 13:07:48.517392 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:48.516920    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7" podUID="0303ad5e-d760-49de-ac87-0719578db8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 07 13:07:49.215755 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:49.215717    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj"]
Sep 07 13:07:49.219474 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:49.219458    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj"
Sep 07 13:07:49.222575 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:49.222545    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 13:07:49.222746 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:49.222545    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flngs\""
Sep 07 13:07:49.229499 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:49.229474    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj"]
Sep 07 13:07:49.323638 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:49.323601    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27b0e8b9-d31c-4258-b07c-30e1409160a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj\" (UID: \"27b0e8b9-d31c-4258-b07c-30e1409160a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj"
Sep 07 13:07:49.425070 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:49.425029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27b0e8b9-d31c-4258-b07c-30e1409160a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj\" (UID: \"27b0e8b9-d31c-4258-b07c-30e1409160a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj"
Sep 07 13:07:49.425740 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:49.425714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27b0e8b9-d31c-4258-b07c-30e1409160a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj\" (UID: \"27b0e8b9-d31c-4258-b07c-30e1409160a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj"
Sep 07 13:07:49.516808 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:49.516711    2565 patch_prober.go:28] interesting pod/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 07 13:07:49.516808 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:49.516779    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7" podUID="0303ad5e-d760-49de-ac87-0719578db8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 07 13:07:49.558639 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:49.558605    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj"
Sep 07 13:07:49.856534 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:49.856509    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj"]
Sep 07 13:07:49.858718 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:07:49.858689    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b0e8b9_d31c_4258_b07c_30e1409160a7.slice/crio-d88e3a6fedd5be83c1c80c6bcccf8286f3fbe55b022f5260ae9b88a11bda8f88 WatchSource:0}: Error finding container d88e3a6fedd5be83c1c80c6bcccf8286f3fbe55b022f5260ae9b88a11bda8f88: Status 404 returned error can't find the container with id d88e3a6fedd5be83c1c80c6bcccf8286f3fbe55b022f5260ae9b88a11bda8f88
Sep 07 13:07:50.516224 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:50.516136    2565 patch_prober.go:28] interesting pod/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 07 13:07:50.516224 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:50.516204    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7" podUID="0303ad5e-d760-49de-ac87-0719578db8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 07 13:07:50.820282 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:50.820243    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj" event={"ID":"27b0e8b9-d31c-4258-b07c-30e1409160a7","Type":"ContainerStarted","Data":"66096f166ebd47530ed92ec378db343a92487c4faed7580c2b69bdab735c238d"}
Sep 07 13:07:50.820746 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:50.820306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj" event={"ID":"27b0e8b9-d31c-4258-b07c-30e1409160a7","Type":"ContainerStarted","Data":"d88e3a6fedd5be83c1c80c6bcccf8286f3fbe55b022f5260ae9b88a11bda8f88"}
Sep 07 13:07:50.838741 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:50.838690    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj" podStartSLOduration=1.8386742470000002 podStartE2EDuration="1.838674247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:07:50.838090976 +0000 UTC m=+887.868957287" watchObservedRunningTime="2026-09-07 13:07:50.838674247 +0000 UTC m=+887.869540580"
Sep 07 13:07:51.008961 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.008914    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"]
Sep 07 13:07:51.013020 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.012999    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.015656 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.015558    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-e9d55a\"/\"kube-root-ca.crt\""
Sep 07 13:07:51.015656 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.015575    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-e9d55a\"/\"openshift-service-ca.crt\""
Sep 07 13:07:51.015656 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.015565    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-e9d55a\"/\"default-dockercfg-65nhq\""
Sep 07 13:07:51.015656 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.015617    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-e9d55a\"/\"autores-model-e9d55a-kserve-self-signed-certs\""
Sep 07 13:07:51.023609 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.023584    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"]
Sep 07 13:07:51.145905 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.145806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-home\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.145905 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.145849    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-dshm\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.146136 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.145939    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe1b549-e519-482c-8c09-b13344437ccd-tls-certs\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.146136 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.146105    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-tmp-dir\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.146249 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.146148    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxk6b\" (UniqueName: \"kubernetes.io/projected/abe1b549-e519-482c-8c09-b13344437ccd-kube-api-access-hxk6b\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.146249 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.146209    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-model-cache\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.247276 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.247237    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-tmp-dir\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.247276 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.247280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxk6b\" (UniqueName: \"kubernetes.io/projected/abe1b549-e519-482c-8c09-b13344437ccd-kube-api-access-hxk6b\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.247574 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.247307    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-model-cache\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.247574 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.247371    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-home\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.247574 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.247389    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-dshm\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.247574 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.247417    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe1b549-e519-482c-8c09-b13344437ccd-tls-certs\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.247774 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.247752    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-tmp-dir\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.247898 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.247879    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-model-cache\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.247990 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.247941    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-home\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.249971 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.249946    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-dshm\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.250499 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.250481    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe1b549-e519-482c-8c09-b13344437ccd-tls-certs\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.265085 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.265056    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxk6b\" (UniqueName: \"kubernetes.io/projected/abe1b549-e519-482c-8c09-b13344437ccd-kube-api-access-hxk6b\") pod \"autores-model-e9d55a-kserve-8598647ff9-6dpmj\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") " pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.323347 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.323294    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:07:51.516843 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.516753    2565 patch_prober.go:28] interesting pod/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 07 13:07:51.516843 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.516822    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7" podUID="0303ad5e-d760-49de-ac87-0719578db8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 07 13:07:51.616117 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.616079    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"]
Sep 07 13:07:51.621528 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:07:51.621495    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabe1b549_e519_482c_8c09_b13344437ccd.slice/crio-e8b55ff8b963e21dd8d32f2d488ae30872c02ce0b0276ee8df5f9e85e56f3506 WatchSource:0}: Error finding container e8b55ff8b963e21dd8d32f2d488ae30872c02ce0b0276ee8df5f9e85e56f3506: Status 404 returned error can't find the container with id e8b55ff8b963e21dd8d32f2d488ae30872c02ce0b0276ee8df5f9e85e56f3506
Sep 07 13:07:51.831478 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.831439    2565 generic.go:356] "Generic (PLEG): container finished" podID="27b0e8b9-d31c-4258-b07c-30e1409160a7" containerID="66096f166ebd47530ed92ec378db343a92487c4faed7580c2b69bdab735c238d" exitCode=0
Sep 07 13:07:51.832055 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.831630    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj" event={"ID":"27b0e8b9-d31c-4258-b07c-30e1409160a7","Type":"ContainerDied","Data":"66096f166ebd47530ed92ec378db343a92487c4faed7580c2b69bdab735c238d"}
Sep 07 13:07:51.834118 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.834089    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj" event={"ID":"abe1b549-e519-482c-8c09-b13344437ccd","Type":"ContainerStarted","Data":"0c00a540e1eb8d31cf904803910a95c43af0802262dca9c481b1de20f0c2b262"}
Sep 07 13:07:51.834268 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.834250    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj" event={"ID":"abe1b549-e519-482c-8c09-b13344437ccd","Type":"ContainerStarted","Data":"e8b55ff8b963e21dd8d32f2d488ae30872c02ce0b0276ee8df5f9e85e56f3506"}
Sep 07 13:07:51.869106 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:51.869041    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj" podStartSLOduration=1.869019537 podStartE2EDuration="1.869019537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:07:51.867361815 +0000 UTC m=+888.898228126" watchObservedRunningTime="2026-09-07 13:07:51.869019537 +0000 UTC m=+888.899885847"
Sep 07 13:07:52.520003 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:52.519975    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:52.520383 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:52.520362    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:52.521098 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:52.521075    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:07:52.976270 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:52.976244    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj"
Sep 07 13:07:53.167052 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:53.166956    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27b0e8b9-d31c-4258-b07c-30e1409160a7-maas-api-service-ca\") pod \"27b0e8b9-d31c-4258-b07c-30e1409160a7\" (UID: \"27b0e8b9-d31c-4258-b07c-30e1409160a7\") "
Sep 07 13:07:53.167363 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:53.167318    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b0e8b9-d31c-4258-b07c-30e1409160a7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "27b0e8b9-d31c-4258-b07c-30e1409160a7" (UID: "27b0e8b9-d31c-4258-b07c-30e1409160a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:07:53.167498 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:53.167480    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27b0e8b9-d31c-4258-b07c-30e1409160a7-maas-api-service-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:07:53.846069 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:53.846041    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj"
Sep 07 13:07:53.846237 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:53.846041    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj" event={"ID":"27b0e8b9-d31c-4258-b07c-30e1409160a7","Type":"ContainerDied","Data":"d88e3a6fedd5be83c1c80c6bcccf8286f3fbe55b022f5260ae9b88a11bda8f88"}
Sep 07 13:07:53.846237 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:53.846169    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d88e3a6fedd5be83c1c80c6bcccf8286f3fbe55b022f5260ae9b88a11bda8f88"
Sep 07 13:07:55.750948 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:55.750850    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"]
Sep 07 13:07:55.751452 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:55.751139    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt" podUID="429e1444-bd0a-4cb3-8efb-596bbdbc8d28" containerName="payload-processing" containerID="cri-o://6da78eafbb87d0a4b880e62e5f62e8301a2ed2aa94166697d242d1877dfb0993" gracePeriod=30
Sep 07 13:07:55.947376 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:55.947302    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj"]
Sep 07 13:07:55.955554 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:55.955527    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c8f5c2a5-l7mtj"]
Sep 07 13:07:56.004672 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.004610    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"
Sep 07 13:07:56.097610 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.097579    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"]
Sep 07 13:07:56.097772 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.097755    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7" podUID="0303ad5e-d760-49de-ac87-0719578db8d6" containerName="istio-proxy" containerID="cri-o://9fefa3739b15252931419d8f10dea9047b873cfa8a2e67788ce984d28bd7fd9d" gracePeriod=30
Sep 07 13:07:56.194785 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.194751    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9z9nn\" (UniqueName: \"kubernetes.io/projected/429e1444-bd0a-4cb3-8efb-596bbdbc8d28-kube-api-access-9z9nn\") pod \"429e1444-bd0a-4cb3-8efb-596bbdbc8d28\" (UID: \"429e1444-bd0a-4cb3-8efb-596bbdbc8d28\") "
Sep 07 13:07:56.195004 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.194851    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/429e1444-bd0a-4cb3-8efb-596bbdbc8d28-plugins-config-volume\") pod \"429e1444-bd0a-4cb3-8efb-596bbdbc8d28\" (UID: \"429e1444-bd0a-4cb3-8efb-596bbdbc8d28\") "
Sep 07 13:07:56.195486 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.195460    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429e1444-bd0a-4cb3-8efb-596bbdbc8d28-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "429e1444-bd0a-4cb3-8efb-596bbdbc8d28" (UID: "429e1444-bd0a-4cb3-8efb-596bbdbc8d28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:07:56.198816 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.198787    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429e1444-bd0a-4cb3-8efb-596bbdbc8d28-kube-api-access-9z9nn" (OuterVolumeSpecName: "kube-api-access-9z9nn") pod "429e1444-bd0a-4cb3-8efb-596bbdbc8d28" (UID: "429e1444-bd0a-4cb3-8efb-596bbdbc8d28"). InnerVolumeSpecName "kube-api-access-9z9nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:07:56.295926 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.295899    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/429e1444-bd0a-4cb3-8efb-596bbdbc8d28-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:07:56.295926 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.295926    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z9nn\" (UniqueName: \"kubernetes.io/projected/429e1444-bd0a-4cb3-8efb-596bbdbc8d28-kube-api-access-9z9nn\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:07:56.865439 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.865401    2565 generic.go:356] "Generic (PLEG): container finished" podID="429e1444-bd0a-4cb3-8efb-596bbdbc8d28" containerID="6da78eafbb87d0a4b880e62e5f62e8301a2ed2aa94166697d242d1877dfb0993" exitCode=0
Sep 07 13:07:56.865439 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.865447    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"
Sep 07 13:07:56.865981 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.865482    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt" event={"ID":"429e1444-bd0a-4cb3-8efb-596bbdbc8d28","Type":"ContainerDied","Data":"6da78eafbb87d0a4b880e62e5f62e8301a2ed2aa94166697d242d1877dfb0993"}
Sep 07 13:07:56.865981 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.865545    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt" event={"ID":"429e1444-bd0a-4cb3-8efb-596bbdbc8d28","Type":"ContainerDied","Data":"50bc8251c39760cd24efe561537d64b8b258e02610c989b0d5bf42f1ceee8944"}
Sep 07 13:07:56.865981 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.865568    2565 scope.go:117] "RemoveContainer" containerID="6da78eafbb87d0a4b880e62e5f62e8301a2ed2aa94166697d242d1877dfb0993"
Sep 07 13:07:56.886707 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.886675    2565 scope.go:117] "RemoveContainer" containerID="6da78eafbb87d0a4b880e62e5f62e8301a2ed2aa94166697d242d1877dfb0993"
Sep 07 13:07:56.887084 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:07:56.887058    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da78eafbb87d0a4b880e62e5f62e8301a2ed2aa94166697d242d1877dfb0993\": container with ID starting with 6da78eafbb87d0a4b880e62e5f62e8301a2ed2aa94166697d242d1877dfb0993 not found: ID does not exist" containerID="6da78eafbb87d0a4b880e62e5f62e8301a2ed2aa94166697d242d1877dfb0993"
Sep 07 13:07:56.887150 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.887096    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da78eafbb87d0a4b880e62e5f62e8301a2ed2aa94166697d242d1877dfb0993"} err="failed to get container status \"6da78eafbb87d0a4b880e62e5f62e8301a2ed2aa94166697d242d1877dfb0993\": rpc error: code = NotFound desc = could not find container \"6da78eafbb87d0a4b880e62e5f62e8301a2ed2aa94166697d242d1877dfb0993\": container with ID starting with 6da78eafbb87d0a4b880e62e5f62e8301a2ed2aa94166697d242d1877dfb0993 not found: ID does not exist"
Sep 07 13:07:56.908918 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.908890    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"]
Sep 07 13:07:56.918507 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:56.918479    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c8f5c2a5-d669f6bb-vwznt"]
Sep 07 13:07:57.489106 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:57.489066    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27b0e8b9-d31c-4258-b07c-30e1409160a7" path="/var/lib/kubelet/pods/27b0e8b9-d31c-4258-b07c-30e1409160a7/volumes"
Sep 07 13:07:57.489495 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:07:57.489481    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="429e1444-bd0a-4cb3-8efb-596bbdbc8d28" path="/var/lib/kubelet/pods/429e1444-bd0a-4cb3-8efb-596bbdbc8d28/volumes"
Sep 07 13:08:01.120147 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.120121    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:08:01.161195 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.161169    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"]
Sep 07 13:08:01.161637 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.161622    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b0e8b9-d31c-4258-b07c-30e1409160a7" containerName="revoke-keys"
Sep 07 13:08:01.161702 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.161639    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b0e8b9-d31c-4258-b07c-30e1409160a7" containerName="revoke-keys"
Sep 07 13:08:01.161702 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.161651    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="429e1444-bd0a-4cb3-8efb-596bbdbc8d28" containerName="payload-processing"
Sep 07 13:08:01.161702 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.161658    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="429e1444-bd0a-4cb3-8efb-596bbdbc8d28" containerName="payload-processing"
Sep 07 13:08:01.161797 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.161759    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="429e1444-bd0a-4cb3-8efb-596bbdbc8d28" containerName="payload-processing"
Sep 07 13:08:01.161797 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.161771    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b0e8b9-d31c-4258-b07c-30e1409160a7" containerName="revoke-keys"
Sep 07 13:08:01.164861 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.164846    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"
Sep 07 13:08:01.167538 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.167520    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-40becd28-dockercfg-cszmw\""
Sep 07 13:08:01.167736 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.167723    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-40becd28\""
Sep 07 13:08:01.180801 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.180770    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"]
Sep 07 13:08:01.250094 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.250054    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1803940-65a3-4ba6-ad8a-e77de5e03d8d-plugins-config-volume\") pod \"payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67\" (UID: \"d1803940-65a3-4ba6-ad8a-e77de5e03d8d\") " pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"
Sep 07 13:08:01.250094 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.250097    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcctq\" (UniqueName: \"kubernetes.io/projected/d1803940-65a3-4ba6-ad8a-e77de5e03d8d-kube-api-access-rcctq\") pod \"payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67\" (UID: \"d1803940-65a3-4ba6-ad8a-e77de5e03d8d\") " pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"
Sep 07 13:08:01.324227 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.324194    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:08:01.324416 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.324364    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:08:01.337684 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.337660    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:08:01.351364 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.351315    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:08:01.351578 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.351559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1803940-65a3-4ba6-ad8a-e77de5e03d8d-plugins-config-volume\") pod \"payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67\" (UID: \"d1803940-65a3-4ba6-ad8a-e77de5e03d8d\") " pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"
Sep 07 13:08:01.351658 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.351612    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcctq\" (UniqueName: \"kubernetes.io/projected/d1803940-65a3-4ba6-ad8a-e77de5e03d8d-kube-api-access-rcctq\") pod \"payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67\" (UID: \"d1803940-65a3-4ba6-ad8a-e77de5e03d8d\") " pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"
Sep 07 13:08:01.352202 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.352183    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1803940-65a3-4ba6-ad8a-e77de5e03d8d-plugins-config-volume\") pod \"payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67\" (UID: \"d1803940-65a3-4ba6-ad8a-e77de5e03d8d\") " pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"
Sep 07 13:08:01.365467 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.365402    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcctq\" (UniqueName: \"kubernetes.io/projected/d1803940-65a3-4ba6-ad8a-e77de5e03d8d-kube-api-access-rcctq\") pod \"payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67\" (UID: \"d1803940-65a3-4ba6-ad8a-e77de5e03d8d\") " pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"
Sep 07 13:08:01.452738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.452702    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0303ad5e-d760-49de-ac87-0719578db8d6-istio-podinfo\") pod \"0303ad5e-d760-49de-ac87-0719578db8d6\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") "
Sep 07 13:08:01.452901 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.452780    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-istio-envoy\") pod \"0303ad5e-d760-49de-ac87-0719578db8d6\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") "
Sep 07 13:08:01.452901 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.452800    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-workload-certs\") pod \"0303ad5e-d760-49de-ac87-0719578db8d6\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") "
Sep 07 13:08:01.452901 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.452842    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0303ad5e-d760-49de-ac87-0719578db8d6-istio-token\") pod \"0303ad5e-d760-49de-ac87-0719578db8d6\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") "
Sep 07 13:08:01.452901 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.452859    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0303ad5e-d760-49de-ac87-0719578db8d6-istiod-ca-cert\") pod \"0303ad5e-d760-49de-ac87-0719578db8d6\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") "
Sep 07 13:08:01.453134 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.452989    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-istio-data\") pod \"0303ad5e-d760-49de-ac87-0719578db8d6\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") "
Sep 07 13:08:01.453134 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.453040    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-credential-socket\") pod \"0303ad5e-d760-49de-ac87-0719578db8d6\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") "
Sep 07 13:08:01.453134 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.453075    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnxgx\" (UniqueName: \"kubernetes.io/projected/0303ad5e-d760-49de-ac87-0719578db8d6-kube-api-access-bnxgx\") pod \"0303ad5e-d760-49de-ac87-0719578db8d6\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") "
Sep 07 13:08:01.453134 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.453110    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-workload-socket\") pod \"0303ad5e-d760-49de-ac87-0719578db8d6\" (UID: \"0303ad5e-d760-49de-ac87-0719578db8d6\") "
Sep 07 13:08:01.453378 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.453151    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0303ad5e-d760-49de-ac87-0719578db8d6" (UID: "0303ad5e-d760-49de-ac87-0719578db8d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:08:01.453378 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.453265    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0303ad5e-d760-49de-ac87-0719578db8d6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0303ad5e-d760-49de-ac87-0719578db8d6" (UID: "0303ad5e-d760-49de-ac87-0719578db8d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:08:01.453378 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.453267    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-istio-data" (OuterVolumeSpecName: "istio-data") pod "0303ad5e-d760-49de-ac87-0719578db8d6" (UID: "0303ad5e-d760-49de-ac87-0719578db8d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:08:01.453564 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.453450    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0303ad5e-d760-49de-ac87-0719578db8d6" (UID: "0303ad5e-d760-49de-ac87-0719578db8d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:08:01.453564 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.453493    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0303ad5e-d760-49de-ac87-0719578db8d6" (UID: "0303ad5e-d760-49de-ac87-0719578db8d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:08:01.453677 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.453596    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-istio-data\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:01.453677 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.453616    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-credential-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:01.453677 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.453634    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-workload-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:01.453677 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.453649    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-workload-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:01.453677 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.453663    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0303ad5e-d760-49de-ac87-0719578db8d6-istiod-ca-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:01.455299 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.455274    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0303ad5e-d760-49de-ac87-0719578db8d6-istio-token" (OuterVolumeSpecName: "istio-token") pod "0303ad5e-d760-49de-ac87-0719578db8d6" (UID: "0303ad5e-d760-49de-ac87-0719578db8d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:08:01.455529 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.455509    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0303ad5e-d760-49de-ac87-0719578db8d6-kube-api-access-bnxgx" (OuterVolumeSpecName: "kube-api-access-bnxgx") pod "0303ad5e-d760-49de-ac87-0719578db8d6" (UID: "0303ad5e-d760-49de-ac87-0719578db8d6"). InnerVolumeSpecName "kube-api-access-bnxgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:08:01.455631 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.455615    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0303ad5e-d760-49de-ac87-0719578db8d6" (UID: "0303ad5e-d760-49de-ac87-0719578db8d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:08:01.455673 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.455628    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0303ad5e-d760-49de-ac87-0719578db8d6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0303ad5e-d760-49de-ac87-0719578db8d6" (UID: "0303ad5e-d760-49de-ac87-0719578db8d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 13:08:01.489063 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.489029    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"
Sep 07 13:08:01.555345 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.555218    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0303ad5e-d760-49de-ac87-0719578db8d6-istio-token\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:01.555345 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.555262    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnxgx\" (UniqueName: \"kubernetes.io/projected/0303ad5e-d760-49de-ac87-0719578db8d6-kube-api-access-bnxgx\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:01.555345 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.555272    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0303ad5e-d760-49de-ac87-0719578db8d6-istio-podinfo\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:01.555345 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.555281    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0303ad5e-d760-49de-ac87-0719578db8d6-istio-envoy\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:01.775690 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.775665    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"]
Sep 07 13:08:01.777877 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:08:01.777842    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1803940_65a3_4ba6_ad8a_e77de5e03d8d.slice/crio-fa962f9729bb0e5728854d111666f14084281562e3123ef5cfe1659c7b8362d7 WatchSource:0}: Error finding container fa962f9729bb0e5728854d111666f14084281562e3123ef5cfe1659c7b8362d7: Status 404 returned error can't find the container with id fa962f9729bb0e5728854d111666f14084281562e3123ef5cfe1659c7b8362d7
Sep 07 13:08:01.895296 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.895260    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67" event={"ID":"d1803940-65a3-4ba6-ad8a-e77de5e03d8d","Type":"ContainerStarted","Data":"d907ec249bfa65f1ec49c07d336df85e31e8a4069244e9c2d846597c02af87a0"}
Sep 07 13:08:01.895510 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.895386    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"
Sep 07 13:08:01.895510 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.895417    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67" event={"ID":"d1803940-65a3-4ba6-ad8a-e77de5e03d8d","Type":"ContainerStarted","Data":"fa962f9729bb0e5728854d111666f14084281562e3123ef5cfe1659c7b8362d7"}
Sep 07 13:08:01.897107 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.897072    2565 generic.go:356] "Generic (PLEG): container finished" podID="0303ad5e-d760-49de-ac87-0719578db8d6" containerID="9fefa3739b15252931419d8f10dea9047b873cfa8a2e67788ce984d28bd7fd9d" exitCode=0
Sep 07 13:08:01.897243 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.897133    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"
Sep 07 13:08:01.897243 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.897157    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7" event={"ID":"0303ad5e-d760-49de-ac87-0719578db8d6","Type":"ContainerDied","Data":"9fefa3739b15252931419d8f10dea9047b873cfa8a2e67788ce984d28bd7fd9d"}
Sep 07 13:08:01.897243 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.897219    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7" event={"ID":"0303ad5e-d760-49de-ac87-0719578db8d6","Type":"ContainerDied","Data":"ec3c482c6b8e0577e69a8813e2a97c1185840e28a94d10e0b75a50ef5fd05e63"}
Sep 07 13:08:01.897434 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.897251    2565 scope.go:117] "RemoveContainer" containerID="9fefa3739b15252931419d8f10dea9047b873cfa8a2e67788ce984d28bd7fd9d"
Sep 07 13:08:01.910020 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.909969    2565 scope.go:117] "RemoveContainer" containerID="0d97f5b8833d86b79ab3f719694121b1cc7bda3eb6140bddbbb9641ec0686b79"
Sep 07 13:08:01.910228 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.910205    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:08:01.919311 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.919289    2565 scope.go:117] "RemoveContainer" containerID="9fefa3739b15252931419d8f10dea9047b873cfa8a2e67788ce984d28bd7fd9d"
Sep 07 13:08:01.919664 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:08:01.919639    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fefa3739b15252931419d8f10dea9047b873cfa8a2e67788ce984d28bd7fd9d\": container with ID starting with 9fefa3739b15252931419d8f10dea9047b873cfa8a2e67788ce984d28bd7fd9d not found: ID does not exist" containerID="9fefa3739b15252931419d8f10dea9047b873cfa8a2e67788ce984d28bd7fd9d"
Sep 07 13:08:01.919738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.919677    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fefa3739b15252931419d8f10dea9047b873cfa8a2e67788ce984d28bd7fd9d"} err="failed to get container status \"9fefa3739b15252931419d8f10dea9047b873cfa8a2e67788ce984d28bd7fd9d\": rpc error: code = NotFound desc = could not find container \"9fefa3739b15252931419d8f10dea9047b873cfa8a2e67788ce984d28bd7fd9d\": container with ID starting with 9fefa3739b15252931419d8f10dea9047b873cfa8a2e67788ce984d28bd7fd9d not found: ID does not exist"
Sep 07 13:08:01.919738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.919705    2565 scope.go:117] "RemoveContainer" containerID="0d97f5b8833d86b79ab3f719694121b1cc7bda3eb6140bddbbb9641ec0686b79"
Sep 07 13:08:01.920003 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:08:01.919985    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d97f5b8833d86b79ab3f719694121b1cc7bda3eb6140bddbbb9641ec0686b79\": container with ID starting with 0d97f5b8833d86b79ab3f719694121b1cc7bda3eb6140bddbbb9641ec0686b79 not found: ID does not exist" containerID="0d97f5b8833d86b79ab3f719694121b1cc7bda3eb6140bddbbb9641ec0686b79"
Sep 07 13:08:01.920075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.920009    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d97f5b8833d86b79ab3f719694121b1cc7bda3eb6140bddbbb9641ec0686b79"} err="failed to get container status \"0d97f5b8833d86b79ab3f719694121b1cc7bda3eb6140bddbbb9641ec0686b79\": rpc error: code = NotFound desc = could not find container \"0d97f5b8833d86b79ab3f719694121b1cc7bda3eb6140bddbbb9641ec0686b79\": container with ID starting with 0d97f5b8833d86b79ab3f719694121b1cc7bda3eb6140bddbbb9641ec0686b79 not found: ID does not exist"
Sep 07 13:08:01.924789 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.924745    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67" podStartSLOduration=0.924732177 podStartE2EDuration="924.732177ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:08:01.923223237 +0000 UTC m=+898.954089547" watchObservedRunningTime="2026-09-07 13:08:01.924732177 +0000 UTC m=+898.955598532"
Sep 07 13:08:01.940413 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.940387    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"]
Sep 07 13:08:01.945909 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:01.945879    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c8f5c2a5-gw-openshift-default-c69f99f96-jlzl7"]
Sep 07 13:08:03.489795 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:03.489761    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0303ad5e-d760-49de-ac87-0719578db8d6" path="/var/lib/kubelet/pods/0303ad5e-d760-49de-ac87-0719578db8d6/volumes"
Sep 07 13:08:12.905986 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:12.905945    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"
Sep 07 13:08:20.307210 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:20.307175    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:08:22.002754 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:22.002718    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd"]
Sep 07 13:08:22.003458 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:22.003436    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0303ad5e-d760-49de-ac87-0719578db8d6" containerName="istio-proxy"
Sep 07 13:08:22.003532 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:22.003464    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0303ad5e-d760-49de-ac87-0719578db8d6" containerName="istio-proxy"
Sep 07 13:08:22.003532 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:22.003486    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0303ad5e-d760-49de-ac87-0719578db8d6" containerName="istio-proxy"
Sep 07 13:08:22.003532 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:22.003495    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0303ad5e-d760-49de-ac87-0719578db8d6" containerName="istio-proxy"
Sep 07 13:08:22.003639 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:22.003612    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0303ad5e-d760-49de-ac87-0719578db8d6" containerName="istio-proxy"
Sep 07 13:08:22.009070 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:22.009047    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd"
Sep 07 13:08:22.011977 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:22.011951    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 13:08:22.012097 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:22.011977    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flngs\""
Sep 07 13:08:22.034840 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:22.034795    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd"]
Sep 07 13:08:22.172051 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:22.172019    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da12efa-7cd9-4846-b64c-2866fd37116b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd\" (UID: \"0da12efa-7cd9-4846-b64c-2866fd37116b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd"
Sep 07 13:08:22.273367 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:22.273241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da12efa-7cd9-4846-b64c-2866fd37116b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd\" (UID: \"0da12efa-7cd9-4846-b64c-2866fd37116b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd"
Sep 07 13:08:22.273931 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:22.273909    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da12efa-7cd9-4846-b64c-2866fd37116b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd\" (UID: \"0da12efa-7cd9-4846-b64c-2866fd37116b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd"
Sep 07 13:08:22.326500 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:22.326465    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd"
Sep 07 13:08:22.604026 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:22.603673    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd"]
Sep 07 13:08:22.607246 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:08:22.607191    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da12efa_7cd9_4846_b64c_2866fd37116b.slice/crio-47bc2df83fd5c281cdbde18a163c7b66d0a8521affe4d24af4681948272aa321 WatchSource:0}: Error finding container 47bc2df83fd5c281cdbde18a163c7b66d0a8521affe4d24af4681948272aa321: Status 404 returned error can't find the container with id 47bc2df83fd5c281cdbde18a163c7b66d0a8521affe4d24af4681948272aa321
Sep 07 13:08:23.012921 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:23.012887    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd" event={"ID":"0da12efa-7cd9-4846-b64c-2866fd37116b","Type":"ContainerStarted","Data":"4b810211d81dd013636e7a1683838ffa738e177765d61641adae973eebcf61c3"}
Sep 07 13:08:23.013314 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:23.012950    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd" event={"ID":"0da12efa-7cd9-4846-b64c-2866fd37116b","Type":"ContainerStarted","Data":"47bc2df83fd5c281cdbde18a163c7b66d0a8521affe4d24af4681948272aa321"}
Sep 07 13:08:23.030741 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:23.030691    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd" podStartSLOduration=2.030675554 podStartE2EDuration="2.030675554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:08:23.028537314 +0000 UTC m=+920.059403625" watchObservedRunningTime="2026-09-07 13:08:23.030675554 +0000 UTC m=+920.061541870"
Sep 07 13:08:24.019745 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:24.019700    2565 generic.go:356] "Generic (PLEG): container finished" podID="0da12efa-7cd9-4846-b64c-2866fd37116b" containerID="4b810211d81dd013636e7a1683838ffa738e177765d61641adae973eebcf61c3" exitCode=0
Sep 07 13:08:24.020134 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:24.019786    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd" event={"ID":"0da12efa-7cd9-4846-b64c-2866fd37116b","Type":"ContainerDied","Data":"4b810211d81dd013636e7a1683838ffa738e177765d61641adae973eebcf61c3"}
Sep 07 13:08:25.162814 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.162787    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd"
Sep 07 13:08:25.202475 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.202444    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da12efa-7cd9-4846-b64c-2866fd37116b-maas-api-service-ca\") pod \"0da12efa-7cd9-4846-b64c-2866fd37116b\" (UID: \"0da12efa-7cd9-4846-b64c-2866fd37116b\") "
Sep 07 13:08:25.202786 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.202763    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da12efa-7cd9-4846-b64c-2866fd37116b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0da12efa-7cd9-4846-b64c-2866fd37116b" (UID: "0da12efa-7cd9-4846-b64c-2866fd37116b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:08:25.303740 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.303710    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da12efa-7cd9-4846-b64c-2866fd37116b-maas-api-service-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:25.503788 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.503756    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"]
Sep 07 13:08:25.504301 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.504277    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da12efa-7cd9-4846-b64c-2866fd37116b" containerName="revoke-keys"
Sep 07 13:08:25.504301 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.504302    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da12efa-7cd9-4846-b64c-2866fd37116b" containerName="revoke-keys"
Sep 07 13:08:25.504494 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.504479    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da12efa-7cd9-4846-b64c-2866fd37116b" containerName="revoke-keys"
Sep 07 13:08:25.504542 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.504499    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0303ad5e-d760-49de-ac87-0719578db8d6" containerName="istio-proxy"
Sep 07 13:08:25.507737 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.507718    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.510625 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.510602    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7ce94d\"/\"openshift-service-ca.crt\""
Sep 07 13:08:25.510728 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.510639    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7ce94d\"/\"kube-root-ca.crt\""
Sep 07 13:08:25.511645 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.511627    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7ce94d\"/\"default-dockercfg-vfvsl\""
Sep 07 13:08:25.511718 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.511647    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7ce94d\"/\"test-model-7ce94d-kserve-self-signed-certs\""
Sep 07 13:08:25.520736 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.520712    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"]
Sep 07 13:08:25.607521 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.607427    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-model-cache\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.607521 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.607474    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-home\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.607521 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.607502    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-tmp-dir\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.607744 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.607563    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d8tn\" (UniqueName: \"kubernetes.io/projected/5e644b85-1278-4c90-b107-ad84fef5e020-kube-api-access-4d8tn\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.607744 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.607620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-dshm\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.607744 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.607655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e644b85-1278-4c90-b107-ad84fef5e020-tls-certs\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.708709 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.708671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e644b85-1278-4c90-b107-ad84fef5e020-tls-certs\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.708873 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.708732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-model-cache\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.708873 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.708766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-home\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.708873 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.708784    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-tmp-dir\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.708873 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.708834    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8tn\" (UniqueName: \"kubernetes.io/projected/5e644b85-1278-4c90-b107-ad84fef5e020-kube-api-access-4d8tn\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.709020 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.708912    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-dshm\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.709251 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.709215    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-home\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.709251 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.709240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-tmp-dir\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.709484 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.709282    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-model-cache\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.711273 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.711254    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-dshm\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.711366 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.711288    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e644b85-1278-4c90-b107-ad84fef5e020-tls-certs\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.717831 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.717812    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d8tn\" (UniqueName: \"kubernetes.io/projected/5e644b85-1278-4c90-b107-ad84fef5e020-kube-api-access-4d8tn\") pod \"test-model-7ce94d-kserve-68bc7c4985-5mqcf\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") " pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:25.817751 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:25.817716    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:26.032067 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:26.032035    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd"
Sep 07 13:08:26.032067 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:26.032045    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd" event={"ID":"0da12efa-7cd9-4846-b64c-2866fd37116b","Type":"ContainerDied","Data":"47bc2df83fd5c281cdbde18a163c7b66d0a8521affe4d24af4681948272aa321"}
Sep 07 13:08:26.032285 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:26.032160    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47bc2df83fd5c281cdbde18a163c7b66d0a8521affe4d24af4681948272aa321"
Sep 07 13:08:26.096374 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:26.096350    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"]
Sep 07 13:08:26.097724 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:08:26.097698    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e644b85_1278_4c90_b107_ad84fef5e020.slice/crio-ba25108ed3d5c313f4d0bfce59171e5af524b17e0642480d2f9e615006c228ce WatchSource:0}: Error finding container ba25108ed3d5c313f4d0bfce59171e5af524b17e0642480d2f9e615006c228ce: Status 404 returned error can't find the container with id ba25108ed3d5c313f4d0bfce59171e5af524b17e0642480d2f9e615006c228ce
Sep 07 13:08:27.038736 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:27.038702    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf" event={"ID":"5e644b85-1278-4c90-b107-ad84fef5e020","Type":"ContainerStarted","Data":"3a36560c687e5b2268c7e9bbd0b5c9d0b2081480dbc85a7c31d022dd22d03608"}
Sep 07 13:08:27.038736 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:27.038743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf" event={"ID":"5e644b85-1278-4c90-b107-ad84fef5e020","Type":"ContainerStarted","Data":"ba25108ed3d5c313f4d0bfce59171e5af524b17e0642480d2f9e615006c228ce"}
Sep 07 13:08:27.059899 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:27.059846    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf" podStartSLOduration=2.059831141 podStartE2EDuration="2.059831141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:08:27.057582257 +0000 UTC m=+924.088448568" watchObservedRunningTime="2026-09-07 13:08:27.059831141 +0000 UTC m=+924.090697450"
Sep 07 13:08:27.710293 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:27.710251    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"]
Sep 07 13:08:29.689783 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:29.689748    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"]
Sep 07 13:08:33.187621 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:33.187582    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd"]
Sep 07 13:08:33.190697 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:33.190670    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e9d55a-6tjmd"]
Sep 07 13:08:33.489055 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:33.488958    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0da12efa-7cd9-4846-b64c-2866fd37116b" path="/var/lib/kubelet/pods/0da12efa-7cd9-4846-b64c-2866fd37116b/volumes"
Sep 07 13:08:35.818745 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:35.818706    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:35.819151 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:35.818760    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:35.831021 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:35.830988    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:36.098865 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:36.098775    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:08:41.312234 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:41.312199    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"]
Sep 07 13:08:41.312731 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:41.312441    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67" podUID="d1803940-65a3-4ba6-ad8a-e77de5e03d8d" containerName="payload-processing" containerID="cri-o://d907ec249bfa65f1ec49c07d336df85e31e8a4069244e9c2d846597c02af87a0" gracePeriod=30
Sep 07 13:08:41.560003 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:41.559970    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"
Sep 07 13:08:41.676218 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:41.676119    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcctq\" (UniqueName: \"kubernetes.io/projected/d1803940-65a3-4ba6-ad8a-e77de5e03d8d-kube-api-access-rcctq\") pod \"d1803940-65a3-4ba6-ad8a-e77de5e03d8d\" (UID: \"d1803940-65a3-4ba6-ad8a-e77de5e03d8d\") "
Sep 07 13:08:41.676396 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:41.676281    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1803940-65a3-4ba6-ad8a-e77de5e03d8d-plugins-config-volume\") pod \"d1803940-65a3-4ba6-ad8a-e77de5e03d8d\" (UID: \"d1803940-65a3-4ba6-ad8a-e77de5e03d8d\") "
Sep 07 13:08:41.676648 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:41.676622    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1803940-65a3-4ba6-ad8a-e77de5e03d8d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1803940-65a3-4ba6-ad8a-e77de5e03d8d" (UID: "d1803940-65a3-4ba6-ad8a-e77de5e03d8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:08:41.678459 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:41.678427    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1803940-65a3-4ba6-ad8a-e77de5e03d8d-kube-api-access-rcctq" (OuterVolumeSpecName: "kube-api-access-rcctq") pod "d1803940-65a3-4ba6-ad8a-e77de5e03d8d" (UID: "d1803940-65a3-4ba6-ad8a-e77de5e03d8d"). InnerVolumeSpecName "kube-api-access-rcctq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:08:41.777710 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:41.777658    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcctq\" (UniqueName: \"kubernetes.io/projected/d1803940-65a3-4ba6-ad8a-e77de5e03d8d-kube-api-access-rcctq\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:41.777710 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:41.777701    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1803940-65a3-4ba6-ad8a-e77de5e03d8d-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:42.126549 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:42.126507    2565 generic.go:356] "Generic (PLEG): container finished" podID="d1803940-65a3-4ba6-ad8a-e77de5e03d8d" containerID="d907ec249bfa65f1ec49c07d336df85e31e8a4069244e9c2d846597c02af87a0" exitCode=0
Sep 07 13:08:42.126719 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:42.126559    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"
Sep 07 13:08:42.126719 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:42.126576    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67" event={"ID":"d1803940-65a3-4ba6-ad8a-e77de5e03d8d","Type":"ContainerDied","Data":"d907ec249bfa65f1ec49c07d336df85e31e8a4069244e9c2d846597c02af87a0"}
Sep 07 13:08:42.126719 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:42.126631    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67" event={"ID":"d1803940-65a3-4ba6-ad8a-e77de5e03d8d","Type":"ContainerDied","Data":"fa962f9729bb0e5728854d111666f14084281562e3123ef5cfe1659c7b8362d7"}
Sep 07 13:08:42.126719 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:42.126662    2565 scope.go:117] "RemoveContainer" containerID="d907ec249bfa65f1ec49c07d336df85e31e8a4069244e9c2d846597c02af87a0"
Sep 07 13:08:42.136044 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:42.136027    2565 scope.go:117] "RemoveContainer" containerID="d907ec249bfa65f1ec49c07d336df85e31e8a4069244e9c2d846597c02af87a0"
Sep 07 13:08:42.136344 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:08:42.136313    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d907ec249bfa65f1ec49c07d336df85e31e8a4069244e9c2d846597c02af87a0\": container with ID starting with d907ec249bfa65f1ec49c07d336df85e31e8a4069244e9c2d846597c02af87a0 not found: ID does not exist" containerID="d907ec249bfa65f1ec49c07d336df85e31e8a4069244e9c2d846597c02af87a0"
Sep 07 13:08:42.136406 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:42.136352    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d907ec249bfa65f1ec49c07d336df85e31e8a4069244e9c2d846597c02af87a0"} err="failed to get container status \"d907ec249bfa65f1ec49c07d336df85e31e8a4069244e9c2d846597c02af87a0\": rpc error: code = NotFound desc = could not find container \"d907ec249bfa65f1ec49c07d336df85e31e8a4069244e9c2d846597c02af87a0\": container with ID starting with d907ec249bfa65f1ec49c07d336df85e31e8a4069244e9c2d846597c02af87a0 not found: ID does not exist"
Sep 07 13:08:42.151131 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:42.151100    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"]
Sep 07 13:08:42.158231 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:42.158203    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40becd28-79bfc5c6dc-2wr67"]
Sep 07 13:08:42.405539 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:42.405458    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:08:42.730615 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:42.730530    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r" podUID="83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d" containerName="main" containerID="cri-o://2e92b6b4a104912fbfce289e63e4056f4276ce12bd842511eb783c275c1fcda2" gracePeriod=45
Sep 07 13:08:42.984438 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:42.982808    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:08:43.090984 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.090941    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-home\") pod \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") "
Sep 07 13:08:43.091174 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.091011    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gr7w\" (UniqueName: \"kubernetes.io/projected/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-kube-api-access-6gr7w\") pod \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") "
Sep 07 13:08:43.091174 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.091095    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-tls-certs\") pod \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") "
Sep 07 13:08:43.091296 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.091235    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-home" (OuterVolumeSpecName: "home") pod "83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d" (UID: "83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:08:43.091296 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.091274    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-model-cache\") pod \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") "
Sep 07 13:08:43.091417 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.091357    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-dshm\") pod \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") "
Sep 07 13:08:43.091477 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.091416    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-tmp-dir\") pod \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\" (UID: \"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d\") "
Sep 07 13:08:43.091572 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.091550    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d" (UID: "83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:08:43.091768 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.091719    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d" (UID: "83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:08:43.092009 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.091794    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-tmp-dir\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:43.092009 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.091810    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-home\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:43.092009 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.091820    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-model-cache\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:43.093639 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.093614    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d" (UID: "83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:08:43.093639 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.093626    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-dshm" (OuterVolumeSpecName: "dshm") pod "83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d" (UID: "83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:08:43.093639 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.093632    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-kube-api-access-6gr7w" (OuterVolumeSpecName: "kube-api-access-6gr7w") pod "83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d" (UID: "83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d"). InnerVolumeSpecName "kube-api-access-6gr7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:08:43.134096 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.134067    2565 generic.go:356] "Generic (PLEG): container finished" podID="83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d" containerID="2e92b6b4a104912fbfce289e63e4056f4276ce12bd842511eb783c275c1fcda2" exitCode=0
Sep 07 13:08:43.134223 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.134102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r" event={"ID":"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d","Type":"ContainerDied","Data":"2e92b6b4a104912fbfce289e63e4056f4276ce12bd842511eb783c275c1fcda2"}
Sep 07 13:08:43.134223 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.134116    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"
Sep 07 13:08:43.134223 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.134126    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r" event={"ID":"83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d","Type":"ContainerDied","Data":"4c69d8633563b385f0822d0e611c5d00dfa9701bab75b8c17826324cc8f5d82b"}
Sep 07 13:08:43.134223 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.134148    2565 scope.go:117] "RemoveContainer" containerID="2e92b6b4a104912fbfce289e63e4056f4276ce12bd842511eb783c275c1fcda2"
Sep 07 13:08:43.143375 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.143285    2565 scope.go:117] "RemoveContainer" containerID="2e92b6b4a104912fbfce289e63e4056f4276ce12bd842511eb783c275c1fcda2"
Sep 07 13:08:43.143841 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:08:43.143644    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e92b6b4a104912fbfce289e63e4056f4276ce12bd842511eb783c275c1fcda2\": container with ID starting with 2e92b6b4a104912fbfce289e63e4056f4276ce12bd842511eb783c275c1fcda2 not found: ID does not exist" containerID="2e92b6b4a104912fbfce289e63e4056f4276ce12bd842511eb783c275c1fcda2"
Sep 07 13:08:43.143841 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.143691    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e92b6b4a104912fbfce289e63e4056f4276ce12bd842511eb783c275c1fcda2"} err="failed to get container status \"2e92b6b4a104912fbfce289e63e4056f4276ce12bd842511eb783c275c1fcda2\": rpc error: code = NotFound desc = could not find container \"2e92b6b4a104912fbfce289e63e4056f4276ce12bd842511eb783c275c1fcda2\": container with ID starting with 2e92b6b4a104912fbfce289e63e4056f4276ce12bd842511eb783c275c1fcda2 not found: ID does not exist"
Sep 07 13:08:43.157225 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.157192    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"]
Sep 07 13:08:43.163686 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.163664    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-c8f26638-kserve-7cbd949549-ngw4r"]
Sep 07 13:08:43.193007 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.192980    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gr7w\" (UniqueName: \"kubernetes.io/projected/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-kube-api-access-6gr7w\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:43.193007 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.193006    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-tls-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:43.193157 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.193019    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d-dshm\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:43.490513 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.490479    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d" path="/var/lib/kubelet/pods/83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d/volumes"
Sep 07 13:08:43.491021 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:43.490999    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1803940-65a3-4ba6-ad8a-e77de5e03d8d" path="/var/lib/kubelet/pods/d1803940-65a3-4ba6-ad8a-e77de5e03d8d/volumes"
Sep 07 13:08:44.709830 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:44.709769    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj" podUID="abe1b549-e519-482c-8c09-b13344437ccd" containerName="main" containerID="cri-o://0c00a540e1eb8d31cf904803910a95c43af0802262dca9c481b1de20f0c2b262" gracePeriod=45
Sep 07 13:08:44.954048 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:44.954019    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:08:45.114424 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.114384    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe1b549-e519-482c-8c09-b13344437ccd-tls-certs\") pod \"abe1b549-e519-482c-8c09-b13344437ccd\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") "
Sep 07 13:08:45.114615 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.114461    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-tmp-dir\") pod \"abe1b549-e519-482c-8c09-b13344437ccd\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") "
Sep 07 13:08:45.114615 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.114483    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-dshm\") pod \"abe1b549-e519-482c-8c09-b13344437ccd\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") "
Sep 07 13:08:45.114615 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.114506    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-home\") pod \"abe1b549-e519-482c-8c09-b13344437ccd\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") "
Sep 07 13:08:45.114615 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.114534    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-model-cache\") pod \"abe1b549-e519-482c-8c09-b13344437ccd\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") "
Sep 07 13:08:45.114615 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.114563    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxk6b\" (UniqueName: \"kubernetes.io/projected/abe1b549-e519-482c-8c09-b13344437ccd-kube-api-access-hxk6b\") pod \"abe1b549-e519-482c-8c09-b13344437ccd\" (UID: \"abe1b549-e519-482c-8c09-b13344437ccd\") "
Sep 07 13:08:45.114868 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.114842    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "abe1b549-e519-482c-8c09-b13344437ccd" (UID: "abe1b549-e519-482c-8c09-b13344437ccd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:08:45.114926 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.114860    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-home" (OuterVolumeSpecName: "home") pod "abe1b549-e519-482c-8c09-b13344437ccd" (UID: "abe1b549-e519-482c-8c09-b13344437ccd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:08:45.114926 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.114873    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-model-cache" (OuterVolumeSpecName: "model-cache") pod "abe1b549-e519-482c-8c09-b13344437ccd" (UID: "abe1b549-e519-482c-8c09-b13344437ccd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:08:45.116915 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.116880    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe1b549-e519-482c-8c09-b13344437ccd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "abe1b549-e519-482c-8c09-b13344437ccd" (UID: "abe1b549-e519-482c-8c09-b13344437ccd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:08:45.117038 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.116949    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-dshm" (OuterVolumeSpecName: "dshm") pod "abe1b549-e519-482c-8c09-b13344437ccd" (UID: "abe1b549-e519-482c-8c09-b13344437ccd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:08:45.117038 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.116979    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe1b549-e519-482c-8c09-b13344437ccd-kube-api-access-hxk6b" (OuterVolumeSpecName: "kube-api-access-hxk6b") pod "abe1b549-e519-482c-8c09-b13344437ccd" (UID: "abe1b549-e519-482c-8c09-b13344437ccd"). InnerVolumeSpecName "kube-api-access-hxk6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:08:45.149968 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.149935    2565 generic.go:356] "Generic (PLEG): container finished" podID="abe1b549-e519-482c-8c09-b13344437ccd" containerID="0c00a540e1eb8d31cf904803910a95c43af0802262dca9c481b1de20f0c2b262" exitCode=0
Sep 07 13:08:45.150140 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.149977    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"
Sep 07 13:08:45.150140 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.149988    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj" event={"ID":"abe1b549-e519-482c-8c09-b13344437ccd","Type":"ContainerDied","Data":"0c00a540e1eb8d31cf904803910a95c43af0802262dca9c481b1de20f0c2b262"}
Sep 07 13:08:45.150140 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.150027    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj" event={"ID":"abe1b549-e519-482c-8c09-b13344437ccd","Type":"ContainerDied","Data":"e8b55ff8b963e21dd8d32f2d488ae30872c02ce0b0276ee8df5f9e85e56f3506"}
Sep 07 13:08:45.150140 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.150049    2565 scope.go:117] "RemoveContainer" containerID="0c00a540e1eb8d31cf904803910a95c43af0802262dca9c481b1de20f0c2b262"
Sep 07 13:08:45.159105 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.159089    2565 scope.go:117] "RemoveContainer" containerID="0c00a540e1eb8d31cf904803910a95c43af0802262dca9c481b1de20f0c2b262"
Sep 07 13:08:45.159458 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:08:45.159433    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c00a540e1eb8d31cf904803910a95c43af0802262dca9c481b1de20f0c2b262\": container with ID starting with 0c00a540e1eb8d31cf904803910a95c43af0802262dca9c481b1de20f0c2b262 not found: ID does not exist" containerID="0c00a540e1eb8d31cf904803910a95c43af0802262dca9c481b1de20f0c2b262"
Sep 07 13:08:45.159538 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.159465    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c00a540e1eb8d31cf904803910a95c43af0802262dca9c481b1de20f0c2b262"} err="failed to get container status \"0c00a540e1eb8d31cf904803910a95c43af0802262dca9c481b1de20f0c2b262\": rpc error: code = NotFound desc = could not find container \"0c00a540e1eb8d31cf904803910a95c43af0802262dca9c481b1de20f0c2b262\": container with ID starting with 0c00a540e1eb8d31cf904803910a95c43af0802262dca9c481b1de20f0c2b262 not found: ID does not exist"
Sep 07 13:08:45.172167 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.172146    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"]
Sep 07 13:08:45.177436 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.177411    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-e9d55a/autores-model-e9d55a-kserve-8598647ff9-6dpmj"]
Sep 07 13:08:45.215775 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.215749    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe1b549-e519-482c-8c09-b13344437ccd-tls-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:45.215775 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.215771    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-tmp-dir\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:45.215775 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.215780    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-dshm\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:45.215940 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.215787    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-home\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:45.215940 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.215795    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe1b549-e519-482c-8c09-b13344437ccd-model-cache\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:45.215940 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.215802    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxk6b\" (UniqueName: \"kubernetes.io/projected/abe1b549-e519-482c-8c09-b13344437ccd-kube-api-access-hxk6b\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:08:45.489086 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:45.489002    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abe1b549-e519-482c-8c09-b13344437ccd" path="/var/lib/kubelet/pods/abe1b549-e519-482c-8c09-b13344437ccd/volumes"
Sep 07 13:08:48.443079 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.443041    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"]
Sep 07 13:08:48.443509 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.443495    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1803940-65a3-4ba6-ad8a-e77de5e03d8d" containerName="payload-processing"
Sep 07 13:08:48.443555 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.443511    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1803940-65a3-4ba6-ad8a-e77de5e03d8d" containerName="payload-processing"
Sep 07 13:08:48.443555 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.443524    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d" containerName="main"
Sep 07 13:08:48.443555 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.443529    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d" containerName="main"
Sep 07 13:08:48.443555 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.443537    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe1b549-e519-482c-8c09-b13344437ccd" containerName="main"
Sep 07 13:08:48.443555 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.443542    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe1b549-e519-482c-8c09-b13344437ccd" containerName="main"
Sep 07 13:08:48.443711 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.443628    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe1b549-e519-482c-8c09-b13344437ccd" containerName="main"
Sep 07 13:08:48.443711 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.443637    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1803940-65a3-4ba6-ad8a-e77de5e03d8d" containerName="payload-processing"
Sep 07 13:08:48.443711 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.443644    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c3810c-a4f0-422a-9f9e-c3eb2e00fc2d" containerName="main"
Sep 07 13:08:48.448046 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.448031    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"
Sep 07 13:08:48.450591 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.450564    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-42173418-dockercfg-zhkdd\""
Sep 07 13:08:48.451011 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.450995    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-42173418\""
Sep 07 13:08:48.465006 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.464982    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"]
Sep 07 13:08:48.550357 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.550286    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56b6162-1e5d-491c-9308-61802881cc2f-plugins-config-volume\") pod \"payload-processing-e2e-derive-42173418-6db69ccb65-q268q\" (UID: \"d56b6162-1e5d-491c-9308-61802881cc2f\") " pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"
Sep 07 13:08:48.550539 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.550374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd2d2\" (UniqueName: \"kubernetes.io/projected/d56b6162-1e5d-491c-9308-61802881cc2f-kube-api-access-hd2d2\") pod \"payload-processing-e2e-derive-42173418-6db69ccb65-q268q\" (UID: \"d56b6162-1e5d-491c-9308-61802881cc2f\") " pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"
Sep 07 13:08:48.651507 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.651464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56b6162-1e5d-491c-9308-61802881cc2f-plugins-config-volume\") pod \"payload-processing-e2e-derive-42173418-6db69ccb65-q268q\" (UID: \"d56b6162-1e5d-491c-9308-61802881cc2f\") " pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"
Sep 07 13:08:48.651699 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.651521    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd2d2\" (UniqueName: \"kubernetes.io/projected/d56b6162-1e5d-491c-9308-61802881cc2f-kube-api-access-hd2d2\") pod \"payload-processing-e2e-derive-42173418-6db69ccb65-q268q\" (UID: \"d56b6162-1e5d-491c-9308-61802881cc2f\") " pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"
Sep 07 13:08:48.652226 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.652201    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56b6162-1e5d-491c-9308-61802881cc2f-plugins-config-volume\") pod \"payload-processing-e2e-derive-42173418-6db69ccb65-q268q\" (UID: \"d56b6162-1e5d-491c-9308-61802881cc2f\") " pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"
Sep 07 13:08:48.661210 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.661180    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd2d2\" (UniqueName: \"kubernetes.io/projected/d56b6162-1e5d-491c-9308-61802881cc2f-kube-api-access-hd2d2\") pod \"payload-processing-e2e-derive-42173418-6db69ccb65-q268q\" (UID: \"d56b6162-1e5d-491c-9308-61802881cc2f\") " pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"
Sep 07 13:08:48.758654 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:48.758564    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"
Sep 07 13:08:49.034607 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:49.034540    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"]
Sep 07 13:08:49.037002 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:08:49.036975    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56b6162_1e5d_491c_9308_61802881cc2f.slice/crio-ea25f008d83666de2d0ffe2b58b1a8012717c16c924b6e0a2be69c81f453bd40 WatchSource:0}: Error finding container ea25f008d83666de2d0ffe2b58b1a8012717c16c924b6e0a2be69c81f453bd40: Status 404 returned error can't find the container with id ea25f008d83666de2d0ffe2b58b1a8012717c16c924b6e0a2be69c81f453bd40
Sep 07 13:08:49.173902 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:49.173860    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q" event={"ID":"d56b6162-1e5d-491c-9308-61802881cc2f","Type":"ContainerStarted","Data":"5379ea7d3b7e601929b1edd02521acf44472f4c43cd012e3897148d4fc1d6db2"}
Sep 07 13:08:49.174080 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:49.173922    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q" event={"ID":"d56b6162-1e5d-491c-9308-61802881cc2f","Type":"ContainerStarted","Data":"ea25f008d83666de2d0ffe2b58b1a8012717c16c924b6e0a2be69c81f453bd40"}
Sep 07 13:08:49.174080 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:49.173959    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"
Sep 07 13:08:49.196847 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:49.196796    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q" podStartSLOduration=1.1967814780000001 podStartE2EDuration="1.196781478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:08:49.195201153 +0000 UTC m=+946.226067484" watchObservedRunningTime="2026-09-07 13:08:49.196781478 +0000 UTC m=+946.227647863"
Sep 07 13:08:56.812647 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:08:56.812613    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:09:00.181620 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:00.181582    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"
Sep 07 13:09:01.883085 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:01.883050    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"]
Sep 07 13:09:01.883514 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:01.883287    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww" podUID="d635af4b-a988-4a4f-9188-24ddcf89ce64" containerName="istio-proxy" containerID="cri-o://3b089c887cef180721eabb2fd79a9af5856b8e1a1eb4b763f0a8d7a4d916137d" gracePeriod=30
Sep 07 13:09:01.973537 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:01.973501    2565 patch_prober.go:28] interesting pod/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 13:09:01.973705 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:01.973578    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww" podUID="d635af4b-a988-4a4f-9188-24ddcf89ce64" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 13:09:07.146578 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.146510    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:09:07.239237 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239205    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d635af4b-a988-4a4f-9188-24ddcf89ce64-istiod-ca-cert\") pod \"d635af4b-a988-4a4f-9188-24ddcf89ce64\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") "
Sep 07 13:09:07.239447 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239251    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-envoy\") pod \"d635af4b-a988-4a4f-9188-24ddcf89ce64\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") "
Sep 07 13:09:07.239447 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239274    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-workload-certs\") pod \"d635af4b-a988-4a4f-9188-24ddcf89ce64\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") "
Sep 07 13:09:07.239447 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239372    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-podinfo\") pod \"d635af4b-a988-4a4f-9188-24ddcf89ce64\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") "
Sep 07 13:09:07.239447 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239412    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-credential-socket\") pod \"d635af4b-a988-4a4f-9188-24ddcf89ce64\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") "
Sep 07 13:09:07.239677 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239449    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-workload-socket\") pod \"d635af4b-a988-4a4f-9188-24ddcf89ce64\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") "
Sep 07 13:09:07.239677 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239496    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-data\") pod \"d635af4b-a988-4a4f-9188-24ddcf89ce64\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") "
Sep 07 13:09:07.239677 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239558    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-token\") pod \"d635af4b-a988-4a4f-9188-24ddcf89ce64\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") "
Sep 07 13:09:07.239677 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239600    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fr44\" (UniqueName: \"kubernetes.io/projected/d635af4b-a988-4a4f-9188-24ddcf89ce64-kube-api-access-7fr44\") pod \"d635af4b-a988-4a4f-9188-24ddcf89ce64\" (UID: \"d635af4b-a988-4a4f-9188-24ddcf89ce64\") "
Sep 07 13:09:07.239677 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239594    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d635af4b-a988-4a4f-9188-24ddcf89ce64" (UID: "d635af4b-a988-4a4f-9188-24ddcf89ce64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:09:07.239930 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239750    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d635af4b-a988-4a4f-9188-24ddcf89ce64-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d635af4b-a988-4a4f-9188-24ddcf89ce64" (UID: "d635af4b-a988-4a4f-9188-24ddcf89ce64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:09:07.239930 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239833    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-data" (OuterVolumeSpecName: "istio-data") pod "d635af4b-a988-4a4f-9188-24ddcf89ce64" (UID: "d635af4b-a988-4a4f-9188-24ddcf89ce64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:09:07.239930 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239876    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d635af4b-a988-4a4f-9188-24ddcf89ce64" (UID: "d635af4b-a988-4a4f-9188-24ddcf89ce64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:09:07.240083 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239953    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-data\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:09:07.240083 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239973    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d635af4b-a988-4a4f-9188-24ddcf89ce64-istiod-ca-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:09:07.240083 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239986    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-workload-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:09:07.240083 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.239999    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-credential-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:09:07.240321 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.240293    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d635af4b-a988-4a4f-9188-24ddcf89ce64" (UID: "d635af4b-a988-4a4f-9188-24ddcf89ce64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:09:07.241713 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.241682    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d635af4b-a988-4a4f-9188-24ddcf89ce64" (UID: "d635af4b-a988-4a4f-9188-24ddcf89ce64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:09:07.241827 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.241757    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d635af4b-a988-4a4f-9188-24ddcf89ce64" (UID: "d635af4b-a988-4a4f-9188-24ddcf89ce64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 13:09:07.241937 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.241920    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d635af4b-a988-4a4f-9188-24ddcf89ce64-kube-api-access-7fr44" (OuterVolumeSpecName: "kube-api-access-7fr44") pod "d635af4b-a988-4a4f-9188-24ddcf89ce64" (UID: "d635af4b-a988-4a4f-9188-24ddcf89ce64"). InnerVolumeSpecName "kube-api-access-7fr44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:09:07.242092 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.242070    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-token" (OuterVolumeSpecName: "istio-token") pod "d635af4b-a988-4a4f-9188-24ddcf89ce64" (UID: "d635af4b-a988-4a4f-9188-24ddcf89ce64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:09:07.273934 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.273898    2565 generic.go:356] "Generic (PLEG): container finished" podID="d635af4b-a988-4a4f-9188-24ddcf89ce64" containerID="3b089c887cef180721eabb2fd79a9af5856b8e1a1eb4b763f0a8d7a4d916137d" exitCode=0
Sep 07 13:09:07.274095 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.273962    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww" event={"ID":"d635af4b-a988-4a4f-9188-24ddcf89ce64","Type":"ContainerDied","Data":"3b089c887cef180721eabb2fd79a9af5856b8e1a1eb4b763f0a8d7a4d916137d"}
Sep 07 13:09:07.274095 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.273969    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"
Sep 07 13:09:07.274095 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.274020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww" event={"ID":"d635af4b-a988-4a4f-9188-24ddcf89ce64","Type":"ContainerDied","Data":"e702d5da7bec090f5fd6825ae0afdac0b6b4b1a8bc69a1ccfca5b297121d05c9"}
Sep 07 13:09:07.274095 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.274044    2565 scope.go:117] "RemoveContainer" containerID="3b089c887cef180721eabb2fd79a9af5856b8e1a1eb4b763f0a8d7a4d916137d"
Sep 07 13:09:07.287778 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.287751    2565 scope.go:117] "RemoveContainer" containerID="10872495be30c240652c098d272d460f242f89a3da5c27134a78ebb2233c5184"
Sep 07 13:09:07.296653 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.296630    2565 scope.go:117] "RemoveContainer" containerID="3b089c887cef180721eabb2fd79a9af5856b8e1a1eb4b763f0a8d7a4d916137d"
Sep 07 13:09:07.296968 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:09:07.296948    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b089c887cef180721eabb2fd79a9af5856b8e1a1eb4b763f0a8d7a4d916137d\": container with ID starting with 3b089c887cef180721eabb2fd79a9af5856b8e1a1eb4b763f0a8d7a4d916137d not found: ID does not exist" containerID="3b089c887cef180721eabb2fd79a9af5856b8e1a1eb4b763f0a8d7a4d916137d"
Sep 07 13:09:07.297034 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.296977    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b089c887cef180721eabb2fd79a9af5856b8e1a1eb4b763f0a8d7a4d916137d"} err="failed to get container status \"3b089c887cef180721eabb2fd79a9af5856b8e1a1eb4b763f0a8d7a4d916137d\": rpc error: code = NotFound desc = could not find container \"3b089c887cef180721eabb2fd79a9af5856b8e1a1eb4b763f0a8d7a4d916137d\": container with ID starting with 3b089c887cef180721eabb2fd79a9af5856b8e1a1eb4b763f0a8d7a4d916137d not found: ID does not exist"
Sep 07 13:09:07.297034 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.296994    2565 scope.go:117] "RemoveContainer" containerID="10872495be30c240652c098d272d460f242f89a3da5c27134a78ebb2233c5184"
Sep 07 13:09:07.297252 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:09:07.297225    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10872495be30c240652c098d272d460f242f89a3da5c27134a78ebb2233c5184\": container with ID starting with 10872495be30c240652c098d272d460f242f89a3da5c27134a78ebb2233c5184 not found: ID does not exist" containerID="10872495be30c240652c098d272d460f242f89a3da5c27134a78ebb2233c5184"
Sep 07 13:09:07.297364 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.297263    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10872495be30c240652c098d272d460f242f89a3da5c27134a78ebb2233c5184"} err="failed to get container status \"10872495be30c240652c098d272d460f242f89a3da5c27134a78ebb2233c5184\": rpc error: code = NotFound desc = could not find container \"10872495be30c240652c098d272d460f242f89a3da5c27134a78ebb2233c5184\": container with ID starting with 10872495be30c240652c098d272d460f242f89a3da5c27134a78ebb2233c5184 not found: ID does not exist"
Sep 07 13:09:07.298530 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.298509    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"]
Sep 07 13:09:07.308163 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.308130    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-e9d55a-openshift-default-7c8c6b9cd4-mm9ww"]
Sep 07 13:09:07.340821 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.340790    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fr44\" (UniqueName: \"kubernetes.io/projected/d635af4b-a988-4a4f-9188-24ddcf89ce64-kube-api-access-7fr44\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:09:07.340821 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.340815    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-envoy\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:09:07.340821 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.340828    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-podinfo\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:09:07.341011 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.340836    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d635af4b-a988-4a4f-9188-24ddcf89ce64-workload-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:09:07.341011 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.340844    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d635af4b-a988-4a4f-9188-24ddcf89ce64-istio-token\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:09:07.488879 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:07.488791    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d635af4b-a988-4a4f-9188-24ddcf89ce64" path="/var/lib/kubelet/pods/d635af4b-a988-4a4f-9188-24ddcf89ce64/volumes"
Sep 07 13:09:08.019154 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.019118    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"]
Sep 07 13:09:08.019562 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.019548    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d635af4b-a988-4a4f-9188-24ddcf89ce64" containerName="istio-proxy"
Sep 07 13:09:08.019623 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.019563    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d635af4b-a988-4a4f-9188-24ddcf89ce64" containerName="istio-proxy"
Sep 07 13:09:08.019623 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.019578    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d635af4b-a988-4a4f-9188-24ddcf89ce64" containerName="istio-proxy"
Sep 07 13:09:08.019623 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.019584    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d635af4b-a988-4a4f-9188-24ddcf89ce64" containerName="istio-proxy"
Sep 07 13:09:08.019736 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.019654    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d635af4b-a988-4a4f-9188-24ddcf89ce64" containerName="istio-proxy"
Sep 07 13:09:08.024205 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.024184    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"
Sep 07 13:09:08.026936 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.026912    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-02ea24\""
Sep 07 13:09:08.039353 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.039297    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"]
Sep 07 13:09:08.039964 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.039947    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d635af4b-a988-4a4f-9188-24ddcf89ce64" containerName="istio-proxy"
Sep 07 13:09:08.042945 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.042926    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"]
Sep 07 13:09:08.043026 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.043013    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"
Sep 07 13:09:08.045447 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.045431    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-02ea24-dockercfg-6sfrl\""
Sep 07 13:09:08.054418 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.054382    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"]
Sep 07 13:09:08.148821 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.148504    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefc3c2a-d0f0-41da-b95f-0868cc21c2c2-plugins-config-volume\") pod \"payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6\" (UID: \"eefc3c2a-d0f0-41da-b95f-0868cc21c2c2\") " pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"
Sep 07 13:09:08.148821 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.148604    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9x2m\" (UniqueName: \"kubernetes.io/projected/f5e8cd24-eab1-4f23-97aa-422cf54e80cb-kube-api-access-z9x2m\") pod \"payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29\" (UID: \"f5e8cd24-eab1-4f23-97aa-422cf54e80cb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"
Sep 07 13:09:08.148821 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.148689    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqgb2\" (UniqueName: \"kubernetes.io/projected/eefc3c2a-d0f0-41da-b95f-0868cc21c2c2-kube-api-access-xqgb2\") pod \"payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6\" (UID: \"eefc3c2a-d0f0-41da-b95f-0868cc21c2c2\") " pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"
Sep 07 13:09:08.148821 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.148730    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5e8cd24-eab1-4f23-97aa-422cf54e80cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29\" (UID: \"f5e8cd24-eab1-4f23-97aa-422cf54e80cb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"
Sep 07 13:09:08.249959 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.249915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqgb2\" (UniqueName: \"kubernetes.io/projected/eefc3c2a-d0f0-41da-b95f-0868cc21c2c2-kube-api-access-xqgb2\") pod \"payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6\" (UID: \"eefc3c2a-d0f0-41da-b95f-0868cc21c2c2\") " pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"
Sep 07 13:09:08.249959 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.249963    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5e8cd24-eab1-4f23-97aa-422cf54e80cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29\" (UID: \"f5e8cd24-eab1-4f23-97aa-422cf54e80cb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"
Sep 07 13:09:08.250213 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.250026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefc3c2a-d0f0-41da-b95f-0868cc21c2c2-plugins-config-volume\") pod \"payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6\" (UID: \"eefc3c2a-d0f0-41da-b95f-0868cc21c2c2\") " pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"
Sep 07 13:09:08.250213 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.250068    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9x2m\" (UniqueName: \"kubernetes.io/projected/f5e8cd24-eab1-4f23-97aa-422cf54e80cb-kube-api-access-z9x2m\") pod \"payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29\" (UID: \"f5e8cd24-eab1-4f23-97aa-422cf54e80cb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"
Sep 07 13:09:08.250807 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.250785    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5e8cd24-eab1-4f23-97aa-422cf54e80cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29\" (UID: \"f5e8cd24-eab1-4f23-97aa-422cf54e80cb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"
Sep 07 13:09:08.250883 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.250786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefc3c2a-d0f0-41da-b95f-0868cc21c2c2-plugins-config-volume\") pod \"payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6\" (UID: \"eefc3c2a-d0f0-41da-b95f-0868cc21c2c2\") " pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"
Sep 07 13:09:08.259037 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.259009    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqgb2\" (UniqueName: \"kubernetes.io/projected/eefc3c2a-d0f0-41da-b95f-0868cc21c2c2-kube-api-access-xqgb2\") pod \"payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6\" (UID: \"eefc3c2a-d0f0-41da-b95f-0868cc21c2c2\") " pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"
Sep 07 13:09:08.259131 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.259080    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9x2m\" (UniqueName: \"kubernetes.io/projected/f5e8cd24-eab1-4f23-97aa-422cf54e80cb-kube-api-access-z9x2m\") pod \"payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29\" (UID: \"f5e8cd24-eab1-4f23-97aa-422cf54e80cb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"
Sep 07 13:09:08.334651 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.334619    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"
Sep 07 13:09:08.355069 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.355041    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"
Sep 07 13:09:08.613697 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.613670    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"]
Sep 07 13:09:08.616017 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:09:08.615987    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e8cd24_eab1_4f23_97aa_422cf54e80cb.slice/crio-bcc876c838e3ec80aabaa7ac633763d574c73a1baabda550499fefcc56f82dac WatchSource:0}: Error finding container bcc876c838e3ec80aabaa7ac633763d574c73a1baabda550499fefcc56f82dac: Status 404 returned error can't find the container with id bcc876c838e3ec80aabaa7ac633763d574c73a1baabda550499fefcc56f82dac
Sep 07 13:09:08.643035 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:08.643008    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"]
Sep 07 13:09:08.649513 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:09:08.649484    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeefc3c2a_d0f0_41da_b95f_0868cc21c2c2.slice/crio-d154decdfc69b6b291f57f09d86b21d53a3be1b24ce8006ef852e92f05fae0d6 WatchSource:0}: Error finding container d154decdfc69b6b291f57f09d86b21d53a3be1b24ce8006ef852e92f05fae0d6: Status 404 returned error can't find the container with id d154decdfc69b6b291f57f09d86b21d53a3be1b24ce8006ef852e92f05fae0d6
Sep 07 13:09:09.287665 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:09.287628    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29" event={"ID":"f5e8cd24-eab1-4f23-97aa-422cf54e80cb","Type":"ContainerStarted","Data":"dc4cf6f97877de3450c96e7ec959dc45055c35af35ebe3dbffe7e9ae56880c49"}
Sep 07 13:09:09.288152 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:09.287696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29" event={"ID":"f5e8cd24-eab1-4f23-97aa-422cf54e80cb","Type":"ContainerStarted","Data":"bcc876c838e3ec80aabaa7ac633763d574c73a1baabda550499fefcc56f82dac"}
Sep 07 13:09:09.288152 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:09.287730    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"
Sep 07 13:09:09.289438 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:09.289412    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6" event={"ID":"eefc3c2a-d0f0-41da-b95f-0868cc21c2c2","Type":"ContainerStarted","Data":"c7497d0a14c30182703654bbdb25e59d9743866f0e9e6bc5033aa8d12cd5b0f6"}
Sep 07 13:09:09.289555 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:09.289460    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6" event={"ID":"eefc3c2a-d0f0-41da-b95f-0868cc21c2c2","Type":"ContainerStarted","Data":"d154decdfc69b6b291f57f09d86b21d53a3be1b24ce8006ef852e92f05fae0d6"}
Sep 07 13:09:09.289555 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:09.289489    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"
Sep 07 13:09:09.309307 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:09.309259    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29" podStartSLOduration=1.309245351 podStartE2EDuration="1.309245351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:09:09.30761102 +0000 UTC m=+966.338477333" watchObservedRunningTime="2026-09-07 13:09:09.309245351 +0000 UTC m=+966.340111661"
Sep 07 13:09:09.327040 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:09.326991    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6" podStartSLOduration=1.3269768659999999 podStartE2EDuration="1.326976866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:09:09.325438611 +0000 UTC m=+966.356304937" watchObservedRunningTime="2026-09-07 13:09:09.326976866 +0000 UTC m=+966.357843221"
Sep 07 13:09:11.208543 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:11.208502    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv"]
Sep 07 13:09:11.212280 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:11.212261    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv"
Sep 07 13:09:11.214789 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:11.214769    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flngs\""
Sep 07 13:09:11.214886 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:11.214783    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 13:09:11.220520 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:11.220245    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv"]
Sep 07 13:09:11.281749 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:11.281711    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2dc7f8-b2d4-4b2a-893c-64a5463aa708-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-42173418-bddlv\" (UID: \"ce2dc7f8-b2d4-4b2a-893c-64a5463aa708\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv"
Sep 07 13:09:11.382502 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:11.382466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2dc7f8-b2d4-4b2a-893c-64a5463aa708-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-42173418-bddlv\" (UID: \"ce2dc7f8-b2d4-4b2a-893c-64a5463aa708\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv"
Sep 07 13:09:11.383068 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:11.383047    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2dc7f8-b2d4-4b2a-893c-64a5463aa708-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-42173418-bddlv\" (UID: \"ce2dc7f8-b2d4-4b2a-893c-64a5463aa708\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv"
Sep 07 13:09:11.524907 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:11.524808    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv"
Sep 07 13:09:11.618741 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:11.618700    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:09:11.803378 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:11.803351    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv"]
Sep 07 13:09:11.805285 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:09:11.805260    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2dc7f8_b2d4_4b2a_893c_64a5463aa708.slice/crio-c84c2b3c8a5e8ae7d0271d56a16c5e1513681b0e5455bfe41d73ff271e01a401 WatchSource:0}: Error finding container c84c2b3c8a5e8ae7d0271d56a16c5e1513681b0e5455bfe41d73ff271e01a401: Status 404 returned error can't find the container with id c84c2b3c8a5e8ae7d0271d56a16c5e1513681b0e5455bfe41d73ff271e01a401
Sep 07 13:09:12.308401 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:12.308363    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv" event={"ID":"ce2dc7f8-b2d4-4b2a-893c-64a5463aa708","Type":"ContainerStarted","Data":"379c89ae336700a86a294ff0ccc275d45e55f35a451649e1144feec86897b718"}
Sep 07 13:09:12.308791 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:12.308426    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv" event={"ID":"ce2dc7f8-b2d4-4b2a-893c-64a5463aa708","Type":"ContainerStarted","Data":"c84c2b3c8a5e8ae7d0271d56a16c5e1513681b0e5455bfe41d73ff271e01a401"}
Sep 07 13:09:12.325911 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:12.325856    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv" podStartSLOduration=1.325837442 podStartE2EDuration="1.325837442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:09:12.325270298 +0000 UTC m=+969.356136624" watchObservedRunningTime="2026-09-07 13:09:12.325837442 +0000 UTC m=+969.356703757"
Sep 07 13:09:13.321089 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:13.321039    2565 generic.go:356] "Generic (PLEG): container finished" podID="ce2dc7f8-b2d4-4b2a-893c-64a5463aa708" containerID="379c89ae336700a86a294ff0ccc275d45e55f35a451649e1144feec86897b718" exitCode=0
Sep 07 13:09:13.321729 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:13.321114    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv" event={"ID":"ce2dc7f8-b2d4-4b2a-893c-64a5463aa708","Type":"ContainerDied","Data":"379c89ae336700a86a294ff0ccc275d45e55f35a451649e1144feec86897b718"}
Sep 07 13:09:14.459982 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:14.459956    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv"
Sep 07 13:09:14.516023 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:14.515993    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2dc7f8-b2d4-4b2a-893c-64a5463aa708-maas-api-service-ca\") pod \"ce2dc7f8-b2d4-4b2a-893c-64a5463aa708\" (UID: \"ce2dc7f8-b2d4-4b2a-893c-64a5463aa708\") "
Sep 07 13:09:14.516395 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:14.516349    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2dc7f8-b2d4-4b2a-893c-64a5463aa708-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ce2dc7f8-b2d4-4b2a-893c-64a5463aa708" (UID: "ce2dc7f8-b2d4-4b2a-893c-64a5463aa708"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:09:14.617820 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:14.617718    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2dc7f8-b2d4-4b2a-893c-64a5463aa708-maas-api-service-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:09:15.339758 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:15.339717    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv" event={"ID":"ce2dc7f8-b2d4-4b2a-893c-64a5463aa708","Type":"ContainerDied","Data":"c84c2b3c8a5e8ae7d0271d56a16c5e1513681b0e5455bfe41d73ff271e01a401"}
Sep 07 13:09:15.339916 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:15.339770    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv"
Sep 07 13:09:15.339916 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:15.339792    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c84c2b3c8a5e8ae7d0271d56a16c5e1513681b0e5455bfe41d73ff271e01a401"
Sep 07 13:09:18.212107 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:18.212072    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"]
Sep 07 13:09:18.212653 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:18.212306    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q" podUID="d56b6162-1e5d-491c-9308-61802881cc2f" containerName="payload-processing" containerID="cri-o://5379ea7d3b7e601929b1edd02521acf44472f4c43cd012e3897148d4fc1d6db2" gracePeriod=30
Sep 07 13:09:18.358841 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:18.358809    2565 generic.go:356] "Generic (PLEG): container finished" podID="d56b6162-1e5d-491c-9308-61802881cc2f" containerID="5379ea7d3b7e601929b1edd02521acf44472f4c43cd012e3897148d4fc1d6db2" exitCode=0
Sep 07 13:09:18.359025 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:18.358881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q" event={"ID":"d56b6162-1e5d-491c-9308-61802881cc2f","Type":"ContainerDied","Data":"5379ea7d3b7e601929b1edd02521acf44472f4c43cd012e3897148d4fc1d6db2"}
Sep 07 13:09:18.400021 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:18.399991    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv"]
Sep 07 13:09:18.402857 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:18.402828    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-42173418-bddlv"]
Sep 07 13:09:18.469982 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:18.469916    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"
Sep 07 13:09:18.562547 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:18.562516    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd2d2\" (UniqueName: \"kubernetes.io/projected/d56b6162-1e5d-491c-9308-61802881cc2f-kube-api-access-hd2d2\") pod \"d56b6162-1e5d-491c-9308-61802881cc2f\" (UID: \"d56b6162-1e5d-491c-9308-61802881cc2f\") "
Sep 07 13:09:18.562714 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:18.562579    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56b6162-1e5d-491c-9308-61802881cc2f-plugins-config-volume\") pod \"d56b6162-1e5d-491c-9308-61802881cc2f\" (UID: \"d56b6162-1e5d-491c-9308-61802881cc2f\") "
Sep 07 13:09:18.562961 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:18.562935    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56b6162-1e5d-491c-9308-61802881cc2f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d56b6162-1e5d-491c-9308-61802881cc2f" (UID: "d56b6162-1e5d-491c-9308-61802881cc2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:09:18.564674 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:18.564648    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56b6162-1e5d-491c-9308-61802881cc2f-kube-api-access-hd2d2" (OuterVolumeSpecName: "kube-api-access-hd2d2") pod "d56b6162-1e5d-491c-9308-61802881cc2f" (UID: "d56b6162-1e5d-491c-9308-61802881cc2f"). InnerVolumeSpecName "kube-api-access-hd2d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:09:18.664557 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:18.664519    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd2d2\" (UniqueName: \"kubernetes.io/projected/d56b6162-1e5d-491c-9308-61802881cc2f-kube-api-access-hd2d2\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:09:18.664557 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:18.664555    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56b6162-1e5d-491c-9308-61802881cc2f-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:09:19.365510 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:19.365473    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q" event={"ID":"d56b6162-1e5d-491c-9308-61802881cc2f","Type":"ContainerDied","Data":"ea25f008d83666de2d0ffe2b58b1a8012717c16c924b6e0a2be69c81f453bd40"}
Sep 07 13:09:19.365510 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:19.365502    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"
Sep 07 13:09:19.366004 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:19.365530    2565 scope.go:117] "RemoveContainer" containerID="5379ea7d3b7e601929b1edd02521acf44472f4c43cd012e3897148d4fc1d6db2"
Sep 07 13:09:19.390678 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:19.390652    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"]
Sep 07 13:09:19.396025 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:19.396001    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-42173418-6db69ccb65-q268q"]
Sep 07 13:09:19.488943 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:19.488904    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce2dc7f8-b2d4-4b2a-893c-64a5463aa708" path="/var/lib/kubelet/pods/ce2dc7f8-b2d4-4b2a-893c-64a5463aa708/volumes"
Sep 07 13:09:19.489472 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:19.489451    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d56b6162-1e5d-491c-9308-61802881cc2f" path="/var/lib/kubelet/pods/d56b6162-1e5d-491c-9308-61802881cc2f/volumes"
Sep 07 13:09:20.297222 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:20.297190    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"
Sep 07 13:09:20.297437 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:20.297379    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"
Sep 07 13:09:29.212837 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.212756    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"]
Sep 07 13:09:29.213209 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.213194    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d56b6162-1e5d-491c-9308-61802881cc2f" containerName="payload-processing"
Sep 07 13:09:29.213254 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.213210    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d56b6162-1e5d-491c-9308-61802881cc2f" containerName="payload-processing"
Sep 07 13:09:29.213254 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.213225    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce2dc7f8-b2d4-4b2a-893c-64a5463aa708" containerName="revoke-keys"
Sep 07 13:09:29.213254 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.213235    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce2dc7f8-b2d4-4b2a-893c-64a5463aa708" containerName="revoke-keys"
Sep 07 13:09:29.213374 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.213339    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d56b6162-1e5d-491c-9308-61802881cc2f" containerName="payload-processing"
Sep 07 13:09:29.213374 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.213351    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce2dc7f8-b2d4-4b2a-893c-64a5463aa708" containerName="revoke-keys"
Sep 07 13:09:29.216244 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.216229    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.218916 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.218893    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-76f028\"/\"openshift-service-ca.crt\""
Sep 07 13:09:29.219968 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.219950    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-76f028\"/\"test-model-76f028-kserve-self-signed-certs\""
Sep 07 13:09:29.220202 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.220177    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-76f028\"/\"kube-root-ca.crt\""
Sep 07 13:09:29.220202 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.220200    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-76f028\"/\"default-dockercfg-54lt2\""
Sep 07 13:09:29.231252 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.231229    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"]
Sep 07 13:09:29.377562 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.377526    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-model-cache\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.377562 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.377564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-home\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.377775 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.377611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-dshm\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.377775 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.377633    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbb76\" (UniqueName: \"kubernetes.io/projected/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-kube-api-access-dbb76\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.377775 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.377714    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-tls-certs\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.377877 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.377829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-tmp-dir\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.478546 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.478464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-tmp-dir\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.478546 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.478503    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-model-cache\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.478546 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.478524    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-home\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.478546 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.478556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-dshm\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.478831 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.478589    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbb76\" (UniqueName: \"kubernetes.io/projected/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-kube-api-access-dbb76\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.478831 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.478681    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-tls-certs\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.478959 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.478884    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-tmp-dir\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.478959 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.478908    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-home\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.479037 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.478966    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-model-cache\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.481203 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.481180    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-dshm\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.481373 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.481358    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-tls-certs\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.487884 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.487856    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbb76\" (UniqueName: \"kubernetes.io/projected/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-kube-api-access-dbb76\") pod \"test-model-76f028-kserve-848f8bd8b7-k8sfb\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") " pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.526779 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.526746    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:29.705779 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.705744    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:09:29.805381 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:29.805322    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"]
Sep 07 13:09:29.808817 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:09:29.808793    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod230ee4e4_d9cf_4cb5_aad3_95b0075ef49f.slice/crio-eb27ccd806d29375eb030795b3ec554884f63b65de2f5964bfec0fd4a5f458dc WatchSource:0}: Error finding container eb27ccd806d29375eb030795b3ec554884f63b65de2f5964bfec0fd4a5f458dc: Status 404 returned error can't find the container with id eb27ccd806d29375eb030795b3ec554884f63b65de2f5964bfec0fd4a5f458dc
Sep 07 13:09:30.429628 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:30.429586    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb" event={"ID":"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f","Type":"ContainerStarted","Data":"4c2bd57c34baa66c765bf9d82a1b406596081fea89f812c1b8a53e9c1c1a9e46"}
Sep 07 13:09:30.429628 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:30.429637    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb" event={"ID":"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f","Type":"ContainerStarted","Data":"eb27ccd806d29375eb030795b3ec554884f63b65de2f5964bfec0fd4a5f458dc"}
Sep 07 13:09:30.449601 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:30.449546    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb" podStartSLOduration=1.449531951 podStartE2EDuration="1.449531951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:09:30.449035355 +0000 UTC m=+987.479901666" watchObservedRunningTime="2026-09-07 13:09:30.449531951 +0000 UTC m=+987.480398260"
Sep 07 13:09:39.527489 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:39.527450    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:39.527953 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:39.527505    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:39.540625 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:39.540599    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:40.492933 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:40.492900    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:09:45.719148 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:09:45.719113    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:10:01.303160 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:01.303124    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:10:04.270775 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.270739    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"]
Sep 07 13:10:04.271210 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.271001    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6" podUID="eefc3c2a-d0f0-41da-b95f-0868cc21c2c2" containerName="payload-processing" containerID="cri-o://c7497d0a14c30182703654bbdb25e59d9743866f0e9e6bc5033aa8d12cd5b0f6" gracePeriod=30
Sep 07 13:10:04.278812 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.278784    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"]
Sep 07 13:10:04.279053 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.279025    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29" podUID="f5e8cd24-eab1-4f23-97aa-422cf54e80cb" containerName="payload-pre-processing" containerID="cri-o://dc4cf6f97877de3450c96e7ec959dc45055c35af35ebe3dbffe7e9ae56880c49" gracePeriod=30
Sep 07 13:10:04.531079 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.531014    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"
Sep 07 13:10:04.553924 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.553898    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"
Sep 07 13:10:04.620526 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.620490    2565 generic.go:356] "Generic (PLEG): container finished" podID="eefc3c2a-d0f0-41da-b95f-0868cc21c2c2" containerID="c7497d0a14c30182703654bbdb25e59d9743866f0e9e6bc5033aa8d12cd5b0f6" exitCode=0
Sep 07 13:10:04.620728 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.620551    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"
Sep 07 13:10:04.620728 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.620574    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6" event={"ID":"eefc3c2a-d0f0-41da-b95f-0868cc21c2c2","Type":"ContainerDied","Data":"c7497d0a14c30182703654bbdb25e59d9743866f0e9e6bc5033aa8d12cd5b0f6"}
Sep 07 13:10:04.620728 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.620656    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6" event={"ID":"eefc3c2a-d0f0-41da-b95f-0868cc21c2c2","Type":"ContainerDied","Data":"d154decdfc69b6b291f57f09d86b21d53a3be1b24ce8006ef852e92f05fae0d6"}
Sep 07 13:10:04.620728 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.620682    2565 scope.go:117] "RemoveContainer" containerID="c7497d0a14c30182703654bbdb25e59d9743866f0e9e6bc5033aa8d12cd5b0f6"
Sep 07 13:10:04.621966 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.621948    2565 generic.go:356] "Generic (PLEG): container finished" podID="f5e8cd24-eab1-4f23-97aa-422cf54e80cb" containerID="dc4cf6f97877de3450c96e7ec959dc45055c35af35ebe3dbffe7e9ae56880c49" exitCode=0
Sep 07 13:10:04.622031 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.621999    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"
Sep 07 13:10:04.622075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.622032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29" event={"ID":"f5e8cd24-eab1-4f23-97aa-422cf54e80cb","Type":"ContainerDied","Data":"dc4cf6f97877de3450c96e7ec959dc45055c35af35ebe3dbffe7e9ae56880c49"}
Sep 07 13:10:04.622121 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.622075    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29" event={"ID":"f5e8cd24-eab1-4f23-97aa-422cf54e80cb","Type":"ContainerDied","Data":"bcc876c838e3ec80aabaa7ac633763d574c73a1baabda550499fefcc56f82dac"}
Sep 07 13:10:04.629564 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.629542    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefc3c2a-d0f0-41da-b95f-0868cc21c2c2-plugins-config-volume\") pod \"eefc3c2a-d0f0-41da-b95f-0868cc21c2c2\" (UID: \"eefc3c2a-d0f0-41da-b95f-0868cc21c2c2\") "
Sep 07 13:10:04.629699 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.629593    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqgb2\" (UniqueName: \"kubernetes.io/projected/eefc3c2a-d0f0-41da-b95f-0868cc21c2c2-kube-api-access-xqgb2\") pod \"eefc3c2a-d0f0-41da-b95f-0868cc21c2c2\" (UID: \"eefc3c2a-d0f0-41da-b95f-0868cc21c2c2\") "
Sep 07 13:10:04.629973 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.629948    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eefc3c2a-d0f0-41da-b95f-0868cc21c2c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eefc3c2a-d0f0-41da-b95f-0868cc21c2c2" (UID: "eefc3c2a-d0f0-41da-b95f-0868cc21c2c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:10:04.630044 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.630026    2565 scope.go:117] "RemoveContainer" containerID="c7497d0a14c30182703654bbdb25e59d9743866f0e9e6bc5033aa8d12cd5b0f6"
Sep 07 13:10:04.630549 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:10:04.630530    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7497d0a14c30182703654bbdb25e59d9743866f0e9e6bc5033aa8d12cd5b0f6\": container with ID starting with c7497d0a14c30182703654bbdb25e59d9743866f0e9e6bc5033aa8d12cd5b0f6 not found: ID does not exist" containerID="c7497d0a14c30182703654bbdb25e59d9743866f0e9e6bc5033aa8d12cd5b0f6"
Sep 07 13:10:04.630600 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.630561    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7497d0a14c30182703654bbdb25e59d9743866f0e9e6bc5033aa8d12cd5b0f6"} err="failed to get container status \"c7497d0a14c30182703654bbdb25e59d9743866f0e9e6bc5033aa8d12cd5b0f6\": rpc error: code = NotFound desc = could not find container \"c7497d0a14c30182703654bbdb25e59d9743866f0e9e6bc5033aa8d12cd5b0f6\": container with ID starting with c7497d0a14c30182703654bbdb25e59d9743866f0e9e6bc5033aa8d12cd5b0f6 not found: ID does not exist"
Sep 07 13:10:04.630600 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.630580    2565 scope.go:117] "RemoveContainer" containerID="dc4cf6f97877de3450c96e7ec959dc45055c35af35ebe3dbffe7e9ae56880c49"
Sep 07 13:10:04.631984 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.631963    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefc3c2a-d0f0-41da-b95f-0868cc21c2c2-kube-api-access-xqgb2" (OuterVolumeSpecName: "kube-api-access-xqgb2") pod "eefc3c2a-d0f0-41da-b95f-0868cc21c2c2" (UID: "eefc3c2a-d0f0-41da-b95f-0868cc21c2c2"). InnerVolumeSpecName "kube-api-access-xqgb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:10:04.639135 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.639120    2565 scope.go:117] "RemoveContainer" containerID="dc4cf6f97877de3450c96e7ec959dc45055c35af35ebe3dbffe7e9ae56880c49"
Sep 07 13:10:04.639417 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:10:04.639395    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc4cf6f97877de3450c96e7ec959dc45055c35af35ebe3dbffe7e9ae56880c49\": container with ID starting with dc4cf6f97877de3450c96e7ec959dc45055c35af35ebe3dbffe7e9ae56880c49 not found: ID does not exist" containerID="dc4cf6f97877de3450c96e7ec959dc45055c35af35ebe3dbffe7e9ae56880c49"
Sep 07 13:10:04.639479 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.639424    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc4cf6f97877de3450c96e7ec959dc45055c35af35ebe3dbffe7e9ae56880c49"} err="failed to get container status \"dc4cf6f97877de3450c96e7ec959dc45055c35af35ebe3dbffe7e9ae56880c49\": rpc error: code = NotFound desc = could not find container \"dc4cf6f97877de3450c96e7ec959dc45055c35af35ebe3dbffe7e9ae56880c49\": container with ID starting with dc4cf6f97877de3450c96e7ec959dc45055c35af35ebe3dbffe7e9ae56880c49 not found: ID does not exist"
Sep 07 13:10:04.730951 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.730912    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5e8cd24-eab1-4f23-97aa-422cf54e80cb-plugins-config-volume\") pod \"f5e8cd24-eab1-4f23-97aa-422cf54e80cb\" (UID: \"f5e8cd24-eab1-4f23-97aa-422cf54e80cb\") "
Sep 07 13:10:04.731128 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.731052    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9x2m\" (UniqueName: \"kubernetes.io/projected/f5e8cd24-eab1-4f23-97aa-422cf54e80cb-kube-api-access-z9x2m\") pod \"f5e8cd24-eab1-4f23-97aa-422cf54e80cb\" (UID: \"f5e8cd24-eab1-4f23-97aa-422cf54e80cb\") "
Sep 07 13:10:04.731402 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.731379    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e8cd24-eab1-4f23-97aa-422cf54e80cb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5e8cd24-eab1-4f23-97aa-422cf54e80cb" (UID: "f5e8cd24-eab1-4f23-97aa-422cf54e80cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:10:04.731508 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.731494    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5e8cd24-eab1-4f23-97aa-422cf54e80cb-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:04.731547 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.731511    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefc3c2a-d0f0-41da-b95f-0868cc21c2c2-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:04.731547 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.731520    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqgb2\" (UniqueName: \"kubernetes.io/projected/eefc3c2a-d0f0-41da-b95f-0868cc21c2c2-kube-api-access-xqgb2\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:04.733318 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.733293    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e8cd24-eab1-4f23-97aa-422cf54e80cb-kube-api-access-z9x2m" (OuterVolumeSpecName: "kube-api-access-z9x2m") pod "f5e8cd24-eab1-4f23-97aa-422cf54e80cb" (UID: "f5e8cd24-eab1-4f23-97aa-422cf54e80cb"). InnerVolumeSpecName "kube-api-access-z9x2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:10:04.832398 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.832360    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9x2m\" (UniqueName: \"kubernetes.io/projected/f5e8cd24-eab1-4f23-97aa-422cf54e80cb-kube-api-access-z9x2m\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:04.949951 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.949903    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"]
Sep 07 13:10:04.952048 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.952022    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-02ea24-678ccfd7c7-q7zc6"]
Sep 07 13:10:04.965885 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.965858    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"]
Sep 07 13:10:04.970267 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:04.970240    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-02ea24-6b7b9fd98-qqz29"]
Sep 07 13:10:05.489006 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:05.488969    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eefc3c2a-d0f0-41da-b95f-0868cc21c2c2" path="/var/lib/kubelet/pods/eefc3c2a-d0f0-41da-b95f-0868cc21c2c2/volumes"
Sep 07 13:10:05.489417 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:05.489363    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5e8cd24-eab1-4f23-97aa-422cf54e80cb" path="/var/lib/kubelet/pods/f5e8cd24-eab1-4f23-97aa-422cf54e80cb/volumes"
Sep 07 13:10:12.435754 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.435715    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"]
Sep 07 13:10:12.436181 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.436161    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e8cd24-eab1-4f23-97aa-422cf54e80cb" containerName="payload-pre-processing"
Sep 07 13:10:12.436181 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.436179    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e8cd24-eab1-4f23-97aa-422cf54e80cb" containerName="payload-pre-processing"
Sep 07 13:10:12.436409 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.436200    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eefc3c2a-d0f0-41da-b95f-0868cc21c2c2" containerName="payload-processing"
Sep 07 13:10:12.436409 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.436209    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eefc3c2a-d0f0-41da-b95f-0868cc21c2c2" containerName="payload-processing"
Sep 07 13:10:12.436409 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.436282    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eefc3c2a-d0f0-41da-b95f-0868cc21c2c2" containerName="payload-processing"
Sep 07 13:10:12.436409 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.436294    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5e8cd24-eab1-4f23-97aa-422cf54e80cb" containerName="payload-pre-processing"
Sep 07 13:10:12.441102 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.441085    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.444084 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.444062    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-3777f2-openshift-default-dockercfg-l8xvq\""
Sep 07 13:10:12.455081 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.455060    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"]
Sep 07 13:10:12.511056 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.511024    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38af0da-d967-45ed-9130-906b9208f6ea-istio-token\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.511219 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.511076    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-istio-envoy\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.511219 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.511108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-credential-socket\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.511219 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.511132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38af0da-d967-45ed-9130-906b9208f6ea-istio-podinfo\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.511219 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.511204    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-istio-data\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.511520 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.511234    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwrxk\" (UniqueName: \"kubernetes.io/projected/a38af0da-d967-45ed-9130-906b9208f6ea-kube-api-access-qwrxk\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.511520 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.511296    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38af0da-d967-45ed-9130-906b9208f6ea-istiod-ca-cert\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.511520 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.511352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-workload-certs\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.511520 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.511387    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-workload-socket\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.612917 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.612877    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-credential-socket\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.612917 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.612918    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38af0da-d967-45ed-9130-906b9208f6ea-istio-podinfo\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.613152 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.612964    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-istio-data\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.613152 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.612989    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwrxk\" (UniqueName: \"kubernetes.io/projected/a38af0da-d967-45ed-9130-906b9208f6ea-kube-api-access-qwrxk\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.613152 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.613027    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38af0da-d967-45ed-9130-906b9208f6ea-istiod-ca-cert\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.613152 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.613053    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-workload-certs\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.613152 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.613091    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-workload-socket\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.613485 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.613153    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38af0da-d967-45ed-9130-906b9208f6ea-istio-token\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.613485 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.613200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-istio-envoy\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.613485 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.613394    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-credential-socket\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.613485 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.613425    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-istio-data\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.613848 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.613821    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-workload-certs\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.614000 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.613854    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38af0da-d967-45ed-9130-906b9208f6ea-istiod-ca-cert\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.614000 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.613938    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-workload-socket\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.615702 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.615676    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38af0da-d967-45ed-9130-906b9208f6ea-istio-podinfo\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.615805 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.615758    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-istio-envoy\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.622524 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.622503    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwrxk\" (UniqueName: \"kubernetes.io/projected/a38af0da-d967-45ed-9130-906b9208f6ea-kube-api-access-qwrxk\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.622991 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.622974    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38af0da-d967-45ed-9130-906b9208f6ea-istio-token\") pod \"e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") " pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:12.751858 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:12.751771    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:13.036203 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:13.036178    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"]
Sep 07 13:10:13.038477 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:10:13.038447    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38af0da_d967_45ed_9130_906b9208f6ea.slice/crio-fe2b39402c5d487762fdeebd4d4ace6beeab8830dd909138be12ee9e1bfa83b6 WatchSource:0}: Error finding container fe2b39402c5d487762fdeebd4d4ace6beeab8830dd909138be12ee9e1bfa83b6: Status 404 returned error can't find the container with id fe2b39402c5d487762fdeebd4d4ace6beeab8830dd909138be12ee9e1bfa83b6
Sep 07 13:10:13.040853 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:13.040819    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:10:13.040964 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:13.040897    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:10:13.040964 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:13.040944    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:10:13.676223 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:13.676178    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml" event={"ID":"a38af0da-d967-45ed-9130-906b9208f6ea","Type":"ContainerStarted","Data":"9548452cfe53041f58463df2231028dde8f45cb0cb0d5605b3eb3a4958ef0c3b"}
Sep 07 13:10:13.676754 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:13.676242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml" event={"ID":"a38af0da-d967-45ed-9130-906b9208f6ea","Type":"ContainerStarted","Data":"fe2b39402c5d487762fdeebd4d4ace6beeab8830dd909138be12ee9e1bfa83b6"}
Sep 07 13:10:13.702315 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:13.702258    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml" podStartSLOduration=1.702242424 podStartE2EDuration="1.702242424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:10:13.700341571 +0000 UTC m=+1030.731207875" watchObservedRunningTime="2026-09-07 13:10:13.702242424 +0000 UTC m=+1030.733108733"
Sep 07 13:10:13.752583 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:13.752554    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:14.757433 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:14.757402    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:15.688654 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:15.688621    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:15.689576 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:15.689553    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:17.129794 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:17.129759    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:10:31.115896 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:31.115858    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"]
Sep 07 13:10:31.116312 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:31.116091    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml" podUID="a38af0da-d967-45ed-9130-906b9208f6ea" containerName="istio-proxy" containerID="cri-o://9548452cfe53041f58463df2231028dde8f45cb0cb0d5605b3eb3a4958ef0c3b" gracePeriod=30
Sep 07 13:10:34.774874 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:34.774826    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"]
Sep 07 13:10:35.872386 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:35.872344    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"]
Sep 07 13:10:36.375484 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.375452    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:36.461370 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.461244    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-istio-envoy\") pod \"a38af0da-d967-45ed-9130-906b9208f6ea\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") "
Sep 07 13:10:36.461370 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.461291    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-workload-socket\") pod \"a38af0da-d967-45ed-9130-906b9208f6ea\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") "
Sep 07 13:10:36.461370 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.461318    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-credential-socket\") pod \"a38af0da-d967-45ed-9130-906b9208f6ea\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") "
Sep 07 13:10:36.461666 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.461440    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-workload-certs\") pod \"a38af0da-d967-45ed-9130-906b9208f6ea\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") "
Sep 07 13:10:36.461666 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.461464    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwrxk\" (UniqueName: \"kubernetes.io/projected/a38af0da-d967-45ed-9130-906b9208f6ea-kube-api-access-qwrxk\") pod \"a38af0da-d967-45ed-9130-906b9208f6ea\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") "
Sep 07 13:10:36.461666 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.461494    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-istio-data\") pod \"a38af0da-d967-45ed-9130-906b9208f6ea\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") "
Sep 07 13:10:36.461666 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.461508    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38af0da-d967-45ed-9130-906b9208f6ea-istio-token\") pod \"a38af0da-d967-45ed-9130-906b9208f6ea\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") "
Sep 07 13:10:36.461666 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.461557    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38af0da-d967-45ed-9130-906b9208f6ea-istio-podinfo\") pod \"a38af0da-d967-45ed-9130-906b9208f6ea\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") "
Sep 07 13:10:36.461666 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.461590    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38af0da-d967-45ed-9130-906b9208f6ea-istiod-ca-cert\") pod \"a38af0da-d967-45ed-9130-906b9208f6ea\" (UID: \"a38af0da-d967-45ed-9130-906b9208f6ea\") "
Sep 07 13:10:36.461666 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.461630    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a38af0da-d967-45ed-9130-906b9208f6ea" (UID: "a38af0da-d967-45ed-9130-906b9208f6ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:10:36.462115 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.461729    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a38af0da-d967-45ed-9130-906b9208f6ea" (UID: "a38af0da-d967-45ed-9130-906b9208f6ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:10:36.462115 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.461855    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a38af0da-d967-45ed-9130-906b9208f6ea" (UID: "a38af0da-d967-45ed-9130-906b9208f6ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:10:36.462115 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.461983    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-istio-data" (OuterVolumeSpecName: "istio-data") pod "a38af0da-d967-45ed-9130-906b9208f6ea" (UID: "a38af0da-d967-45ed-9130-906b9208f6ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:10:36.462115 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.462061    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38af0da-d967-45ed-9130-906b9208f6ea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a38af0da-d967-45ed-9130-906b9208f6ea" (UID: "a38af0da-d967-45ed-9130-906b9208f6ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:10:36.462520 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.462195    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-workload-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:36.462520 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.462212    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-istio-data\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:36.462520 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.462224    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-workload-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:36.462520 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.462232    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-credential-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:36.463858 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.463830    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a38af0da-d967-45ed-9130-906b9208f6ea" (UID: "a38af0da-d967-45ed-9130-906b9208f6ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:10:36.463992 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.463858    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38af0da-d967-45ed-9130-906b9208f6ea-istio-token" (OuterVolumeSpecName: "istio-token") pod "a38af0da-d967-45ed-9130-906b9208f6ea" (UID: "a38af0da-d967-45ed-9130-906b9208f6ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:10:36.463992 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.463944    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a38af0da-d967-45ed-9130-906b9208f6ea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a38af0da-d967-45ed-9130-906b9208f6ea" (UID: "a38af0da-d967-45ed-9130-906b9208f6ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 13:10:36.464247 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.464227    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38af0da-d967-45ed-9130-906b9208f6ea-kube-api-access-qwrxk" (OuterVolumeSpecName: "kube-api-access-qwrxk") pod "a38af0da-d967-45ed-9130-906b9208f6ea" (UID: "a38af0da-d967-45ed-9130-906b9208f6ea"). InnerVolumeSpecName "kube-api-access-qwrxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:10:36.563347 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.563300    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwrxk\" (UniqueName: \"kubernetes.io/projected/a38af0da-d967-45ed-9130-906b9208f6ea-kube-api-access-qwrxk\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:36.563347 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.563349    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a38af0da-d967-45ed-9130-906b9208f6ea-istio-token\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:36.563591 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.563362    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a38af0da-d967-45ed-9130-906b9208f6ea-istio-podinfo\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:36.563591 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.563375    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a38af0da-d967-45ed-9130-906b9208f6ea-istiod-ca-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:36.563591 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.563386    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a38af0da-d967-45ed-9130-906b9208f6ea-istio-envoy\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:36.808159 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.808121    2565 generic.go:356] "Generic (PLEG): container finished" podID="a38af0da-d967-45ed-9130-906b9208f6ea" containerID="9548452cfe53041f58463df2231028dde8f45cb0cb0d5605b3eb3a4958ef0c3b" exitCode=0
Sep 07 13:10:36.808342 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.808175    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"
Sep 07 13:10:36.808342 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.808213    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml" event={"ID":"a38af0da-d967-45ed-9130-906b9208f6ea","Type":"ContainerDied","Data":"9548452cfe53041f58463df2231028dde8f45cb0cb0d5605b3eb3a4958ef0c3b"}
Sep 07 13:10:36.808342 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.808270    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml" event={"ID":"a38af0da-d967-45ed-9130-906b9208f6ea","Type":"ContainerDied","Data":"fe2b39402c5d487762fdeebd4d4ace6beeab8830dd909138be12ee9e1bfa83b6"}
Sep 07 13:10:36.808342 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.808294    2565 scope.go:117] "RemoveContainer" containerID="9548452cfe53041f58463df2231028dde8f45cb0cb0d5605b3eb3a4958ef0c3b"
Sep 07 13:10:36.820176 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.820159    2565 scope.go:117] "RemoveContainer" containerID="9548452cfe53041f58463df2231028dde8f45cb0cb0d5605b3eb3a4958ef0c3b"
Sep 07 13:10:36.820482 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:10:36.820462    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9548452cfe53041f58463df2231028dde8f45cb0cb0d5605b3eb3a4958ef0c3b\": container with ID starting with 9548452cfe53041f58463df2231028dde8f45cb0cb0d5605b3eb3a4958ef0c3b not found: ID does not exist" containerID="9548452cfe53041f58463df2231028dde8f45cb0cb0d5605b3eb3a4958ef0c3b"
Sep 07 13:10:36.820559 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.820489    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9548452cfe53041f58463df2231028dde8f45cb0cb0d5605b3eb3a4958ef0c3b"} err="failed to get container status \"9548452cfe53041f58463df2231028dde8f45cb0cb0d5605b3eb3a4958ef0c3b\": rpc error: code = NotFound desc = could not find container \"9548452cfe53041f58463df2231028dde8f45cb0cb0d5605b3eb3a4958ef0c3b\": container with ID starting with 9548452cfe53041f58463df2231028dde8f45cb0cb0d5605b3eb3a4958ef0c3b not found: ID does not exist"
Sep 07 13:10:36.834436 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.834413    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"]
Sep 07 13:10:36.839918 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:36.839893    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-3777f2-openshift-default-546cb5cc44-6l5ml"]
Sep 07 13:10:37.489405 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:37.489373    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a38af0da-d967-45ed-9130-906b9208f6ea" path="/var/lib/kubelet/pods/a38af0da-d967-45ed-9130-906b9208f6ea/volumes"
Sep 07 13:10:38.005303 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:38.005274    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:10:49.613295 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:49.613259    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:10:49.795693 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:49.795652    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf" podUID="5e644b85-1278-4c90-b107-ad84fef5e020" containerName="main" containerID="cri-o://3a36560c687e5b2268c7e9bbd0b5c9d0b2081480dbc85a7c31d022dd22d03608" gracePeriod=45
Sep 07 13:10:50.057170 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.057142    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:10:50.101211 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.101177    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-model-cache\") pod \"5e644b85-1278-4c90-b107-ad84fef5e020\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") "
Sep 07 13:10:50.101387 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.101243    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-tmp-dir\") pod \"5e644b85-1278-4c90-b107-ad84fef5e020\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") "
Sep 07 13:10:50.101387 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.101278    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d8tn\" (UniqueName: \"kubernetes.io/projected/5e644b85-1278-4c90-b107-ad84fef5e020-kube-api-access-4d8tn\") pod \"5e644b85-1278-4c90-b107-ad84fef5e020\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") "
Sep 07 13:10:50.101387 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.101306    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-home\") pod \"5e644b85-1278-4c90-b107-ad84fef5e020\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") "
Sep 07 13:10:50.101387 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.101369    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e644b85-1278-4c90-b107-ad84fef5e020-tls-certs\") pod \"5e644b85-1278-4c90-b107-ad84fef5e020\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") "
Sep 07 13:10:50.101613 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.101409    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-dshm\") pod \"5e644b85-1278-4c90-b107-ad84fef5e020\" (UID: \"5e644b85-1278-4c90-b107-ad84fef5e020\") "
Sep 07 13:10:50.101613 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.101495    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e644b85-1278-4c90-b107-ad84fef5e020" (UID: "5e644b85-1278-4c90-b107-ad84fef5e020"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:10:50.101613 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.101599    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e644b85-1278-4c90-b107-ad84fef5e020" (UID: "5e644b85-1278-4c90-b107-ad84fef5e020"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:10:50.101764 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.101655    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-home" (OuterVolumeSpecName: "home") pod "5e644b85-1278-4c90-b107-ad84fef5e020" (UID: "5e644b85-1278-4c90-b107-ad84fef5e020"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:10:50.101764 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.101733    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-model-cache\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:50.101764 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.101753    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-tmp-dir\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:50.103717 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.103687    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e644b85-1278-4c90-b107-ad84fef5e020-kube-api-access-4d8tn" (OuterVolumeSpecName: "kube-api-access-4d8tn") pod "5e644b85-1278-4c90-b107-ad84fef5e020" (UID: "5e644b85-1278-4c90-b107-ad84fef5e020"). InnerVolumeSpecName "kube-api-access-4d8tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:10:50.104150 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.104129    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-dshm" (OuterVolumeSpecName: "dshm") pod "5e644b85-1278-4c90-b107-ad84fef5e020" (UID: "5e644b85-1278-4c90-b107-ad84fef5e020"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:10:50.104220 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.104142    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e644b85-1278-4c90-b107-ad84fef5e020-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e644b85-1278-4c90-b107-ad84fef5e020" (UID: "5e644b85-1278-4c90-b107-ad84fef5e020"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:10:50.202865 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.202778    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d8tn\" (UniqueName: \"kubernetes.io/projected/5e644b85-1278-4c90-b107-ad84fef5e020-kube-api-access-4d8tn\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:50.202865 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.202810    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-home\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:50.202865 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.202820    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e644b85-1278-4c90-b107-ad84fef5e020-tls-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:50.202865 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.202829    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e644b85-1278-4c90-b107-ad84fef5e020-dshm\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:50.891583 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.891549    2565 generic.go:356] "Generic (PLEG): container finished" podID="5e644b85-1278-4c90-b107-ad84fef5e020" containerID="3a36560c687e5b2268c7e9bbd0b5c9d0b2081480dbc85a7c31d022dd22d03608" exitCode=0
Sep 07 13:10:50.892110 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.891606    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"
Sep 07 13:10:50.892110 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.891646    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf" event={"ID":"5e644b85-1278-4c90-b107-ad84fef5e020","Type":"ContainerDied","Data":"3a36560c687e5b2268c7e9bbd0b5c9d0b2081480dbc85a7c31d022dd22d03608"}
Sep 07 13:10:50.892110 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.891702    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf" event={"ID":"5e644b85-1278-4c90-b107-ad84fef5e020","Type":"ContainerDied","Data":"ba25108ed3d5c313f4d0bfce59171e5af524b17e0642480d2f9e615006c228ce"}
Sep 07 13:10:50.892110 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.891729    2565 scope.go:117] "RemoveContainer" containerID="3a36560c687e5b2268c7e9bbd0b5c9d0b2081480dbc85a7c31d022dd22d03608"
Sep 07 13:10:50.893147 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.893120    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb" podUID="230ee4e4-d9cf-4cb5-aad3-95b0075ef49f" containerName="main" containerID="cri-o://4c2bd57c34baa66c765bf9d82a1b406596081fea89f812c1b8a53e9c1c1a9e46" gracePeriod=45
Sep 07 13:10:50.904972 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.904951    2565 scope.go:117] "RemoveContainer" containerID="3a36560c687e5b2268c7e9bbd0b5c9d0b2081480dbc85a7c31d022dd22d03608"
Sep 07 13:10:50.906066 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:10:50.906037    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a36560c687e5b2268c7e9bbd0b5c9d0b2081480dbc85a7c31d022dd22d03608\": container with ID starting with 3a36560c687e5b2268c7e9bbd0b5c9d0b2081480dbc85a7c31d022dd22d03608 not found: ID does not exist" containerID="3a36560c687e5b2268c7e9bbd0b5c9d0b2081480dbc85a7c31d022dd22d03608"
Sep 07 13:10:50.906229 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.906075    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a36560c687e5b2268c7e9bbd0b5c9d0b2081480dbc85a7c31d022dd22d03608"} err="failed to get container status \"3a36560c687e5b2268c7e9bbd0b5c9d0b2081480dbc85a7c31d022dd22d03608\": rpc error: code = NotFound desc = could not find container \"3a36560c687e5b2268c7e9bbd0b5c9d0b2081480dbc85a7c31d022dd22d03608\": container with ID starting with 3a36560c687e5b2268c7e9bbd0b5c9d0b2081480dbc85a7c31d022dd22d03608 not found: ID does not exist"
Sep 07 13:10:50.919503 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.919476    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"]
Sep 07 13:10:50.924423 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:50.924400    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7ce94d/test-model-7ce94d-kserve-68bc7c4985-5mqcf"]
Sep 07 13:10:51.140298 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.140271    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:10:51.212563 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.212461    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-tmp-dir\") pod \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") "
Sep 07 13:10:51.212563 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.212510    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbb76\" (UniqueName: \"kubernetes.io/projected/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-kube-api-access-dbb76\") pod \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") "
Sep 07 13:10:51.212563 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.212538    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-home\") pod \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") "
Sep 07 13:10:51.212841 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.212623    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-dshm\") pod \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") "
Sep 07 13:10:51.212841 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.212679    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-tls-certs\") pod \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") "
Sep 07 13:10:51.212841 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.212722    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-model-cache\") pod \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\" (UID: \"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f\") "
Sep 07 13:10:51.212841 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.212738    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "230ee4e4-d9cf-4cb5-aad3-95b0075ef49f" (UID: "230ee4e4-d9cf-4cb5-aad3-95b0075ef49f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:10:51.213019 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.212873    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-home" (OuterVolumeSpecName: "home") pod "230ee4e4-d9cf-4cb5-aad3-95b0075ef49f" (UID: "230ee4e4-d9cf-4cb5-aad3-95b0075ef49f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:10:51.213060 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.213016    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-model-cache" (OuterVolumeSpecName: "model-cache") pod "230ee4e4-d9cf-4cb5-aad3-95b0075ef49f" (UID: "230ee4e4-d9cf-4cb5-aad3-95b0075ef49f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:10:51.213115 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.213099    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-model-cache\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:51.213159 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.213120    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-tmp-dir\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:51.213159 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.213129    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-home\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:51.215025 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.214998    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "230ee4e4-d9cf-4cb5-aad3-95b0075ef49f" (UID: "230ee4e4-d9cf-4cb5-aad3-95b0075ef49f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:10:51.215353 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.215298    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-dshm" (OuterVolumeSpecName: "dshm") pod "230ee4e4-d9cf-4cb5-aad3-95b0075ef49f" (UID: "230ee4e4-d9cf-4cb5-aad3-95b0075ef49f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:10:51.215353 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.215304    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-kube-api-access-dbb76" (OuterVolumeSpecName: "kube-api-access-dbb76") pod "230ee4e4-d9cf-4cb5-aad3-95b0075ef49f" (UID: "230ee4e4-d9cf-4cb5-aad3-95b0075ef49f"). InnerVolumeSpecName "kube-api-access-dbb76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:10:51.313910 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.313874    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbb76\" (UniqueName: \"kubernetes.io/projected/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-kube-api-access-dbb76\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:51.313910 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.313905    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-dshm\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:51.313910 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.313916    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f-tls-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:10:51.489870 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.489790    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e644b85-1278-4c90-b107-ad84fef5e020" path="/var/lib/kubelet/pods/5e644b85-1278-4c90-b107-ad84fef5e020/volumes"
Sep 07 13:10:51.899846 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.899811    2565 generic.go:356] "Generic (PLEG): container finished" podID="230ee4e4-d9cf-4cb5-aad3-95b0075ef49f" containerID="4c2bd57c34baa66c765bf9d82a1b406596081fea89f812c1b8a53e9c1c1a9e46" exitCode=0
Sep 07 13:10:51.899846 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.899851    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"
Sep 07 13:10:51.900419 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.899851    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb" event={"ID":"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f","Type":"ContainerDied","Data":"4c2bd57c34baa66c765bf9d82a1b406596081fea89f812c1b8a53e9c1c1a9e46"}
Sep 07 13:10:51.900419 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.899912    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb" event={"ID":"230ee4e4-d9cf-4cb5-aad3-95b0075ef49f","Type":"ContainerDied","Data":"eb27ccd806d29375eb030795b3ec554884f63b65de2f5964bfec0fd4a5f458dc"}
Sep 07 13:10:51.900419 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.899934    2565 scope.go:117] "RemoveContainer" containerID="4c2bd57c34baa66c765bf9d82a1b406596081fea89f812c1b8a53e9c1c1a9e46"
Sep 07 13:10:51.908977 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.908962    2565 scope.go:117] "RemoveContainer" containerID="4c2bd57c34baa66c765bf9d82a1b406596081fea89f812c1b8a53e9c1c1a9e46"
Sep 07 13:10:51.909255 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:10:51.909232    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2bd57c34baa66c765bf9d82a1b406596081fea89f812c1b8a53e9c1c1a9e46\": container with ID starting with 4c2bd57c34baa66c765bf9d82a1b406596081fea89f812c1b8a53e9c1c1a9e46 not found: ID does not exist" containerID="4c2bd57c34baa66c765bf9d82a1b406596081fea89f812c1b8a53e9c1c1a9e46"
Sep 07 13:10:51.909317 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.909263    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2bd57c34baa66c765bf9d82a1b406596081fea89f812c1b8a53e9c1c1a9e46"} err="failed to get container status \"4c2bd57c34baa66c765bf9d82a1b406596081fea89f812c1b8a53e9c1c1a9e46\": rpc error: code = NotFound desc = could not find container \"4c2bd57c34baa66c765bf9d82a1b406596081fea89f812c1b8a53e9c1c1a9e46\": container with ID starting with 4c2bd57c34baa66c765bf9d82a1b406596081fea89f812c1b8a53e9c1c1a9e46 not found: ID does not exist"
Sep 07 13:10:51.918886 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.918860    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"]
Sep 07 13:10:51.923091 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:51.923070    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-76f028/test-model-76f028-kserve-848f8bd8b7-k8sfb"]
Sep 07 13:10:53.487767 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:53.487723    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="230ee4e4-d9cf-4cb5-aad3-95b0075ef49f" path="/var/lib/kubelet/pods/230ee4e4-d9cf-4cb5-aad3-95b0075ef49f/volumes"
Sep 07 13:10:54.612346 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:54.612303    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"]
Sep 07 13:10:54.612744 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:54.612534    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc" podUID="4e53887c-3654-486d-85a6-191e7dbea57a" containerName="istio-proxy" containerID="cri-o://733e3e90f13b3d79ba6128865edeba6b79c56f859640e8e90c8f29517e06887b" gracePeriod=30
Sep 07 13:10:55.098852 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:55.098815    2565 patch_prober.go:28] interesting pod/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 13:10:55.099022 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:55.098886    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc" podUID="4e53887c-3654-486d-85a6-191e7dbea57a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 13:10:58.232826 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:58.232792    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:10:59.865501 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.865471    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:10:59.943882 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.943789    2565 generic.go:356] "Generic (PLEG): container finished" podID="4e53887c-3654-486d-85a6-191e7dbea57a" containerID="733e3e90f13b3d79ba6128865edeba6b79c56f859640e8e90c8f29517e06887b" exitCode=0
Sep 07 13:10:59.943882 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.943840    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc" event={"ID":"4e53887c-3654-486d-85a6-191e7dbea57a","Type":"ContainerDied","Data":"733e3e90f13b3d79ba6128865edeba6b79c56f859640e8e90c8f29517e06887b"}
Sep 07 13:10:59.944079 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.943892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc" event={"ID":"4e53887c-3654-486d-85a6-191e7dbea57a","Type":"ContainerDied","Data":"2c1276f7052fc6b2ae02ebd24870cb9af32a7ec0873553af93b43871f0471142"}
Sep 07 13:10:59.944079 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.943916    2565 scope.go:117] "RemoveContainer" containerID="733e3e90f13b3d79ba6128865edeba6b79c56f859640e8e90c8f29517e06887b"
Sep 07 13:10:59.944079 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.943847    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"
Sep 07 13:10:59.952778 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.952760    2565 scope.go:117] "RemoveContainer" containerID="608bb006486854f9704fc397e8f7832d748669fd60a7f773a4a179bdf8fe743e"
Sep 07 13:10:59.960628 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.960607    2565 scope.go:117] "RemoveContainer" containerID="733e3e90f13b3d79ba6128865edeba6b79c56f859640e8e90c8f29517e06887b"
Sep 07 13:10:59.960911 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:10:59.960888    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733e3e90f13b3d79ba6128865edeba6b79c56f859640e8e90c8f29517e06887b\": container with ID starting with 733e3e90f13b3d79ba6128865edeba6b79c56f859640e8e90c8f29517e06887b not found: ID does not exist" containerID="733e3e90f13b3d79ba6128865edeba6b79c56f859640e8e90c8f29517e06887b"
Sep 07 13:10:59.960980 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.960925    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733e3e90f13b3d79ba6128865edeba6b79c56f859640e8e90c8f29517e06887b"} err="failed to get container status \"733e3e90f13b3d79ba6128865edeba6b79c56f859640e8e90c8f29517e06887b\": rpc error: code = NotFound desc = could not find container \"733e3e90f13b3d79ba6128865edeba6b79c56f859640e8e90c8f29517e06887b\": container with ID starting with 733e3e90f13b3d79ba6128865edeba6b79c56f859640e8e90c8f29517e06887b not found: ID does not exist"
Sep 07 13:10:59.960980 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.960948    2565 scope.go:117] "RemoveContainer" containerID="608bb006486854f9704fc397e8f7832d748669fd60a7f773a4a179bdf8fe743e"
Sep 07 13:10:59.961221 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:10:59.961203    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608bb006486854f9704fc397e8f7832d748669fd60a7f773a4a179bdf8fe743e\": container with ID starting with 608bb006486854f9704fc397e8f7832d748669fd60a7f773a4a179bdf8fe743e not found: ID does not exist" containerID="608bb006486854f9704fc397e8f7832d748669fd60a7f773a4a179bdf8fe743e"
Sep 07 13:10:59.961268 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.961228    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608bb006486854f9704fc397e8f7832d748669fd60a7f773a4a179bdf8fe743e"} err="failed to get container status \"608bb006486854f9704fc397e8f7832d748669fd60a7f773a4a179bdf8fe743e\": rpc error: code = NotFound desc = could not find container \"608bb006486854f9704fc397e8f7832d748669fd60a7f773a4a179bdf8fe743e\": container with ID starting with 608bb006486854f9704fc397e8f7832d748669fd60a7f773a4a179bdf8fe743e not found: ID does not exist"
Sep 07 13:10:59.999620 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.999589    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-istio-envoy\") pod \"4e53887c-3654-486d-85a6-191e7dbea57a\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") "
Sep 07 13:10:59.999767 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.999664    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-workload-socket\") pod \"4e53887c-3654-486d-85a6-191e7dbea57a\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") "
Sep 07 13:10:59.999767 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.999698    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw5fq\" (UniqueName: \"kubernetes.io/projected/4e53887c-3654-486d-85a6-191e7dbea57a-kube-api-access-hw5fq\") pod \"4e53887c-3654-486d-85a6-191e7dbea57a\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") "
Sep 07 13:10:59.999767 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.999717    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-credential-socket\") pod \"4e53887c-3654-486d-85a6-191e7dbea57a\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") "
Sep 07 13:10:59.999767 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.999739    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-workload-certs\") pod \"4e53887c-3654-486d-85a6-191e7dbea57a\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") "
Sep 07 13:10:59.999767 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.999760    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e53887c-3654-486d-85a6-191e7dbea57a-istio-token\") pod \"4e53887c-3654-486d-85a6-191e7dbea57a\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") "
Sep 07 13:11:00.000034 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.999827    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e53887c-3654-486d-85a6-191e7dbea57a-istio-podinfo\") pod \"4e53887c-3654-486d-85a6-191e7dbea57a\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") "
Sep 07 13:11:00.000034 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.999869    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-istio-data\") pod \"4e53887c-3654-486d-85a6-191e7dbea57a\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") "
Sep 07 13:11:00.000034 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.999913    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e53887c-3654-486d-85a6-191e7dbea57a-istiod-ca-cert\") pod \"4e53887c-3654-486d-85a6-191e7dbea57a\" (UID: \"4e53887c-3654-486d-85a6-191e7dbea57a\") "
Sep 07 13:11:00.000034 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:10:59.999958    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4e53887c-3654-486d-85a6-191e7dbea57a" (UID: "4e53887c-3654-486d-85a6-191e7dbea57a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:11:00.000236 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.000066    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4e53887c-3654-486d-85a6-191e7dbea57a" (UID: "4e53887c-3654-486d-85a6-191e7dbea57a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:11:00.000236 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.000085    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4e53887c-3654-486d-85a6-191e7dbea57a" (UID: "4e53887c-3654-486d-85a6-191e7dbea57a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:11:00.000236 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.000225    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-istio-data" (OuterVolumeSpecName: "istio-data") pod "4e53887c-3654-486d-85a6-191e7dbea57a" (UID: "4e53887c-3654-486d-85a6-191e7dbea57a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:11:00.000594 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.000382    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-workload-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:00.000594 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.000390    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e53887c-3654-486d-85a6-191e7dbea57a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4e53887c-3654-486d-85a6-191e7dbea57a" (UID: "4e53887c-3654-486d-85a6-191e7dbea57a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:11:00.000594 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.000399    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-credential-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:00.000594 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.000409    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-workload-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:00.000594 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.000418    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-istio-data\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:00.002028 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.002000    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4e53887c-3654-486d-85a6-191e7dbea57a" (UID: "4e53887c-3654-486d-85a6-191e7dbea57a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:11:00.002028 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.002014    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e53887c-3654-486d-85a6-191e7dbea57a-kube-api-access-hw5fq" (OuterVolumeSpecName: "kube-api-access-hw5fq") pod "4e53887c-3654-486d-85a6-191e7dbea57a" (UID: "4e53887c-3654-486d-85a6-191e7dbea57a"). InnerVolumeSpecName "kube-api-access-hw5fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:11:00.002470 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.002450    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e53887c-3654-486d-85a6-191e7dbea57a-istio-token" (OuterVolumeSpecName: "istio-token") pod "4e53887c-3654-486d-85a6-191e7dbea57a" (UID: "4e53887c-3654-486d-85a6-191e7dbea57a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:11:00.002669 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.002651    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e53887c-3654-486d-85a6-191e7dbea57a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4e53887c-3654-486d-85a6-191e7dbea57a" (UID: "4e53887c-3654-486d-85a6-191e7dbea57a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 13:11:00.101116 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.101067    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw5fq\" (UniqueName: \"kubernetes.io/projected/4e53887c-3654-486d-85a6-191e7dbea57a-kube-api-access-hw5fq\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:00.101116 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.101113    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e53887c-3654-486d-85a6-191e7dbea57a-istio-token\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:00.101116 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.101123    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e53887c-3654-486d-85a6-191e7dbea57a-istio-podinfo\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:00.101116 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.101131    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e53887c-3654-486d-85a6-191e7dbea57a-istiod-ca-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:00.101425 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.101139    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e53887c-3654-486d-85a6-191e7dbea57a-istio-envoy\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:00.268198 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.268160    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"]
Sep 07 13:11:00.274755 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:00.274729    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7ce94d-openshift-default-75cfb98c78-xtptc"]
Sep 07 13:11:01.488442 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:01.488406    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e53887c-3654-486d-85a6-191e7dbea57a" path="/var/lib/kubelet/pods/4e53887c-3654-486d-85a6-191e7dbea57a/volumes"
Sep 07 13:11:05.762186 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.762148    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"]
Sep 07 13:11:05.762738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.762600    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e644b85-1278-4c90-b107-ad84fef5e020" containerName="main"
Sep 07 13:11:05.762738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.762613    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e644b85-1278-4c90-b107-ad84fef5e020" containerName="main"
Sep 07 13:11:05.762738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.762623    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a38af0da-d967-45ed-9130-906b9208f6ea" containerName="istio-proxy"
Sep 07 13:11:05.762738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.762629    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a38af0da-d967-45ed-9130-906b9208f6ea" containerName="istio-proxy"
Sep 07 13:11:05.762738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.762637    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230ee4e4-d9cf-4cb5-aad3-95b0075ef49f" containerName="main"
Sep 07 13:11:05.762738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.762643    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="230ee4e4-d9cf-4cb5-aad3-95b0075ef49f" containerName="main"
Sep 07 13:11:05.762738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.762650    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e53887c-3654-486d-85a6-191e7dbea57a" containerName="istio-proxy"
Sep 07 13:11:05.762738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.762655    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e53887c-3654-486d-85a6-191e7dbea57a" containerName="istio-proxy"
Sep 07 13:11:05.762738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.762663    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e53887c-3654-486d-85a6-191e7dbea57a" containerName="istio-proxy"
Sep 07 13:11:05.762738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.762668    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e53887c-3654-486d-85a6-191e7dbea57a" containerName="istio-proxy"
Sep 07 13:11:05.762738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.762727    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e644b85-1278-4c90-b107-ad84fef5e020" containerName="main"
Sep 07 13:11:05.762738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.762734    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e53887c-3654-486d-85a6-191e7dbea57a" containerName="istio-proxy"
Sep 07 13:11:05.762738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.762742    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e53887c-3654-486d-85a6-191e7dbea57a" containerName="istio-proxy"
Sep 07 13:11:05.763149 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.762749    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a38af0da-d967-45ed-9130-906b9208f6ea" containerName="istio-proxy"
Sep 07 13:11:05.763149 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.762757    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="230ee4e4-d9cf-4cb5-aad3-95b0075ef49f" containerName="main"
Sep 07 13:11:05.767695 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.767679    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"
Sep 07 13:11:05.770505 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.770483    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-953bf16f\""
Sep 07 13:11:05.779959 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.779934    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"]
Sep 07 13:11:05.839197 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.839166    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"]
Sep 07 13:11:05.842860 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.842839    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"
Sep 07 13:11:05.845542 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.845514    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-953bf16f-dockercfg-qgvr7\""
Sep 07 13:11:05.857212 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.857189    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"]
Sep 07 13:11:05.859854 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.859830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6mbb\" (UniqueName: \"kubernetes.io/projected/d7fa12ae-eeca-46ef-b0d3-a1e0b5126337-kube-api-access-z6mbb\") pod \"payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp\" (UID: \"d7fa12ae-eeca-46ef-b0d3-a1e0b5126337\") " pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"
Sep 07 13:11:05.859965 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.859870    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7fa12ae-eeca-46ef-b0d3-a1e0b5126337-plugins-config-volume\") pod \"payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp\" (UID: \"d7fa12ae-eeca-46ef-b0d3-a1e0b5126337\") " pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"
Sep 07 13:11:05.961388 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.961350    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6mbb\" (UniqueName: \"kubernetes.io/projected/d7fa12ae-eeca-46ef-b0d3-a1e0b5126337-kube-api-access-z6mbb\") pod \"payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp\" (UID: \"d7fa12ae-eeca-46ef-b0d3-a1e0b5126337\") " pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"
Sep 07 13:11:05.961578 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.961400    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7fa12ae-eeca-46ef-b0d3-a1e0b5126337-plugins-config-volume\") pod \"payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp\" (UID: \"d7fa12ae-eeca-46ef-b0d3-a1e0b5126337\") " pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"
Sep 07 13:11:05.961578 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.961429    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptkz7\" (UniqueName: \"kubernetes.io/projected/f03b3760-9702-4d75-bedd-f337b7454f6f-kube-api-access-ptkz7\") pod \"payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4\" (UID: \"f03b3760-9702-4d75-bedd-f337b7454f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"
Sep 07 13:11:05.961578 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.961509    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03b3760-9702-4d75-bedd-f337b7454f6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4\" (UID: \"f03b3760-9702-4d75-bedd-f337b7454f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"
Sep 07 13:11:05.962166 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.962141    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7fa12ae-eeca-46ef-b0d3-a1e0b5126337-plugins-config-volume\") pod \"payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp\" (UID: \"d7fa12ae-eeca-46ef-b0d3-a1e0b5126337\") " pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"
Sep 07 13:11:05.977197 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:05.977166    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6mbb\" (UniqueName: \"kubernetes.io/projected/d7fa12ae-eeca-46ef-b0d3-a1e0b5126337-kube-api-access-z6mbb\") pod \"payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp\" (UID: \"d7fa12ae-eeca-46ef-b0d3-a1e0b5126337\") " pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"
Sep 07 13:11:06.062814 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:06.062767    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03b3760-9702-4d75-bedd-f337b7454f6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4\" (UID: \"f03b3760-9702-4d75-bedd-f337b7454f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"
Sep 07 13:11:06.062989 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:06.062892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptkz7\" (UniqueName: \"kubernetes.io/projected/f03b3760-9702-4d75-bedd-f337b7454f6f-kube-api-access-ptkz7\") pod \"payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4\" (UID: \"f03b3760-9702-4d75-bedd-f337b7454f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"
Sep 07 13:11:06.063677 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:06.063655    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03b3760-9702-4d75-bedd-f337b7454f6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4\" (UID: \"f03b3760-9702-4d75-bedd-f337b7454f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"
Sep 07 13:11:06.074775 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:06.074748    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptkz7\" (UniqueName: \"kubernetes.io/projected/f03b3760-9702-4d75-bedd-f337b7454f6f-kube-api-access-ptkz7\") pod \"payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4\" (UID: \"f03b3760-9702-4d75-bedd-f337b7454f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"
Sep 07 13:11:06.078684 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:06.078652    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"
Sep 07 13:11:06.152842 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:06.152813    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"
Sep 07 13:11:06.371770 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:06.371680    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"]
Sep 07 13:11:06.374448 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:11:06.374419    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7fa12ae_eeca_46ef_b0d3_a1e0b5126337.slice/crio-91fb5f5197cf8c2eab4bd0fde9c26bd578932bd632602bc52a67a05c77ff741d WatchSource:0}: Error finding container 91fb5f5197cf8c2eab4bd0fde9c26bd578932bd632602bc52a67a05c77ff741d: Status 404 returned error can't find the container with id 91fb5f5197cf8c2eab4bd0fde9c26bd578932bd632602bc52a67a05c77ff741d
Sep 07 13:11:06.439790 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:06.439755    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"]
Sep 07 13:11:06.440678 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:11:06.440600    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03b3760_9702_4d75_bedd_f337b7454f6f.slice/crio-2aec21abd7938b7686e6d1f52bdce48581b8d9d881dca6da8e4e4124774f1e37 WatchSource:0}: Error finding container 2aec21abd7938b7686e6d1f52bdce48581b8d9d881dca6da8e4e4124774f1e37: Status 404 returned error can't find the container with id 2aec21abd7938b7686e6d1f52bdce48581b8d9d881dca6da8e4e4124774f1e37
Sep 07 13:11:06.988711 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:06.988677    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp" event={"ID":"d7fa12ae-eeca-46ef-b0d3-a1e0b5126337","Type":"ContainerStarted","Data":"bed2dc8d254af81f51e046b5d0e869a93c75229e9e6d25666e09c0deddbcbef3"}
Sep 07 13:11:06.989158 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:06.988730    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp" event={"ID":"d7fa12ae-eeca-46ef-b0d3-a1e0b5126337","Type":"ContainerStarted","Data":"91fb5f5197cf8c2eab4bd0fde9c26bd578932bd632602bc52a67a05c77ff741d"}
Sep 07 13:11:06.989158 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:06.988779    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"
Sep 07 13:11:06.990849 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:06.990823    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4" event={"ID":"f03b3760-9702-4d75-bedd-f337b7454f6f","Type":"ContainerStarted","Data":"531bccd0af5fb405450d46e735af843ac3c6d4012f028b48bbfe4e1377d120d9"}
Sep 07 13:11:06.990960 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:06.990867    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4" event={"ID":"f03b3760-9702-4d75-bedd-f337b7454f6f","Type":"ContainerStarted","Data":"2aec21abd7938b7686e6d1f52bdce48581b8d9d881dca6da8e4e4124774f1e37"}
Sep 07 13:11:06.991001 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:06.990969    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"
Sep 07 13:11:07.012118 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:07.012069    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp" podStartSLOduration=2.012053035 podStartE2EDuration="2.012053035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:11:07.011205046 +0000 UTC m=+1084.042071355" watchObservedRunningTime="2026-09-07 13:11:07.012053035 +0000 UTC m=+1084.042919344"
Sep 07 13:11:07.032373 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:07.032306    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4" podStartSLOduration=2.032291553 podStartE2EDuration="2.032291553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:11:07.030672688 +0000 UTC m=+1084.061538998" watchObservedRunningTime="2026-09-07 13:11:07.032291553 +0000 UTC m=+1084.063157862"
Sep 07 13:11:09.213395 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:09.213355    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:11:13.972791 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:13.972761    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"]
Sep 07 13:11:13.973185 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:13.972982    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29" podUID="30ece672-58d2-489b-a652-9308ee1960d5" containerName="istio-proxy" containerID="cri-o://e2fb0187527a4bf473d2d3db76ab857a79953ee25b6964538497a8448d487f2c" gracePeriod=30
Sep 07 13:11:17.998354 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:17.998291    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"
Sep 07 13:11:17.998707 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:17.998504    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"
Sep 07 13:11:19.224904 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.224878    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:11:19.300620 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.300588    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ece672-58d2-489b-a652-9308ee1960d5-istio-podinfo\") pod \"30ece672-58d2-489b-a652-9308ee1960d5\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") "
Sep 07 13:11:19.300817 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.300647    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ece672-58d2-489b-a652-9308ee1960d5-istio-token\") pod \"30ece672-58d2-489b-a652-9308ee1960d5\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") "
Sep 07 13:11:19.300817 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.300678    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-istio-data\") pod \"30ece672-58d2-489b-a652-9308ee1960d5\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") "
Sep 07 13:11:19.300817 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.300701    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk2nd\" (UniqueName: \"kubernetes.io/projected/30ece672-58d2-489b-a652-9308ee1960d5-kube-api-access-qk2nd\") pod \"30ece672-58d2-489b-a652-9308ee1960d5\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") "
Sep 07 13:11:19.300817 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.300722    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-workload-socket\") pod \"30ece672-58d2-489b-a652-9308ee1960d5\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") "
Sep 07 13:11:19.300817 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.300737    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-workload-certs\") pod \"30ece672-58d2-489b-a652-9308ee1960d5\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") "
Sep 07 13:11:19.300817 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.300759    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-istio-envoy\") pod \"30ece672-58d2-489b-a652-9308ee1960d5\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") "
Sep 07 13:11:19.301148 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.300982    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ece672-58d2-489b-a652-9308ee1960d5-istiod-ca-cert\") pod \"30ece672-58d2-489b-a652-9308ee1960d5\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") "
Sep 07 13:11:19.301148 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.301012    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-istio-data" (OuterVolumeSpecName: "istio-data") pod "30ece672-58d2-489b-a652-9308ee1960d5" (UID: "30ece672-58d2-489b-a652-9308ee1960d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:11:19.301148 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.301075    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-credential-socket\") pod \"30ece672-58d2-489b-a652-9308ee1960d5\" (UID: \"30ece672-58d2-489b-a652-9308ee1960d5\") "
Sep 07 13:11:19.301148 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.301099    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "30ece672-58d2-489b-a652-9308ee1960d5" (UID: "30ece672-58d2-489b-a652-9308ee1960d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:11:19.301564 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.301292    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "30ece672-58d2-489b-a652-9308ee1960d5" (UID: "30ece672-58d2-489b-a652-9308ee1960d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:11:19.301564 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.301387    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "30ece672-58d2-489b-a652-9308ee1960d5" (UID: "30ece672-58d2-489b-a652-9308ee1960d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:11:19.301564 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.301431    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ece672-58d2-489b-a652-9308ee1960d5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "30ece672-58d2-489b-a652-9308ee1960d5" (UID: "30ece672-58d2-489b-a652-9308ee1960d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:11:19.301564 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.301486    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-workload-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:19.301564 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.301503    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-workload-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:19.301564 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.301516    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-credential-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:19.301564 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.301528    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-istio-data\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:19.303045 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.303019    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30ece672-58d2-489b-a652-9308ee1960d5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "30ece672-58d2-489b-a652-9308ee1960d5" (UID: "30ece672-58d2-489b-a652-9308ee1960d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 13:11:19.303308 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.303280    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ece672-58d2-489b-a652-9308ee1960d5-kube-api-access-qk2nd" (OuterVolumeSpecName: "kube-api-access-qk2nd") pod "30ece672-58d2-489b-a652-9308ee1960d5" (UID: "30ece672-58d2-489b-a652-9308ee1960d5"). InnerVolumeSpecName "kube-api-access-qk2nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:11:19.303547 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.303529    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "30ece672-58d2-489b-a652-9308ee1960d5" (UID: "30ece672-58d2-489b-a652-9308ee1960d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:11:19.303547 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.303539    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ece672-58d2-489b-a652-9308ee1960d5-istio-token" (OuterVolumeSpecName: "istio-token") pod "30ece672-58d2-489b-a652-9308ee1960d5" (UID: "30ece672-58d2-489b-a652-9308ee1960d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:11:19.402510 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.402471    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ece672-58d2-489b-a652-9308ee1960d5-istio-podinfo\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:19.402510 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.402501    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ece672-58d2-489b-a652-9308ee1960d5-istio-token\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:19.402510 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.402509    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk2nd\" (UniqueName: \"kubernetes.io/projected/30ece672-58d2-489b-a652-9308ee1960d5-kube-api-access-qk2nd\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:19.402510 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.402518    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ece672-58d2-489b-a652-9308ee1960d5-istio-envoy\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:19.402780 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.402527    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ece672-58d2-489b-a652-9308ee1960d5-istiod-ca-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:19.779348 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.779253    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"]
Sep 07 13:11:19.779724 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.779709    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ece672-58d2-489b-a652-9308ee1960d5" containerName="istio-proxy"
Sep 07 13:11:19.779764 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.779726    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ece672-58d2-489b-a652-9308ee1960d5" containerName="istio-proxy"
Sep 07 13:11:19.779764 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.779750    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ece672-58d2-489b-a652-9308ee1960d5" containerName="istio-proxy"
Sep 07 13:11:19.779764 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.779756    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ece672-58d2-489b-a652-9308ee1960d5" containerName="istio-proxy"
Sep 07 13:11:19.779860 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.779815    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ece672-58d2-489b-a652-9308ee1960d5" containerName="istio-proxy"
Sep 07 13:11:19.782813 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.782796    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"
Sep 07 13:11:19.785503 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.785475    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-aa813e\""
Sep 07 13:11:19.805910 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.805884    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"]
Sep 07 13:11:19.807336 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.807308    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh9sl\" (UniqueName: \"kubernetes.io/projected/6e21fb80-e2d9-425e-b336-449536271a12-kube-api-access-xh9sl\") pod \"payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh\" (UID: \"6e21fb80-e2d9-425e-b336-449536271a12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"
Sep 07 13:11:19.807455 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.807371    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e21fb80-e2d9-425e-b336-449536271a12-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh\" (UID: \"6e21fb80-e2d9-425e-b336-449536271a12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"
Sep 07 13:11:19.908585 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.908540    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh9sl\" (UniqueName: \"kubernetes.io/projected/6e21fb80-e2d9-425e-b336-449536271a12-kube-api-access-xh9sl\") pod \"payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh\" (UID: \"6e21fb80-e2d9-425e-b336-449536271a12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"
Sep 07 13:11:19.908770 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.908606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e21fb80-e2d9-425e-b336-449536271a12-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh\" (UID: \"6e21fb80-e2d9-425e-b336-449536271a12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"
Sep 07 13:11:19.909344 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.909306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e21fb80-e2d9-425e-b336-449536271a12-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh\" (UID: \"6e21fb80-e2d9-425e-b336-449536271a12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"
Sep 07 13:11:19.922498 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:19.922474    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh9sl\" (UniqueName: \"kubernetes.io/projected/6e21fb80-e2d9-425e-b336-449536271a12-kube-api-access-xh9sl\") pod \"payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh\" (UID: \"6e21fb80-e2d9-425e-b336-449536271a12\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"
Sep 07 13:11:20.068424 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:20.068391    2565 generic.go:356] "Generic (PLEG): container finished" podID="30ece672-58d2-489b-a652-9308ee1960d5" containerID="e2fb0187527a4bf473d2d3db76ab857a79953ee25b6964538497a8448d487f2c" exitCode=0
Sep 07 13:11:20.068609 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:20.068449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29" event={"ID":"30ece672-58d2-489b-a652-9308ee1960d5","Type":"ContainerDied","Data":"e2fb0187527a4bf473d2d3db76ab857a79953ee25b6964538497a8448d487f2c"}
Sep 07 13:11:20.068609 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:20.068505    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29" event={"ID":"30ece672-58d2-489b-a652-9308ee1960d5","Type":"ContainerDied","Data":"0f3be5cc9c4a763db05f3b84561ff84e31c05937a8b2973026e42a311e6d838e"}
Sep 07 13:11:20.068609 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:20.068520    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"
Sep 07 13:11:20.068609 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:20.068535    2565 scope.go:117] "RemoveContainer" containerID="e2fb0187527a4bf473d2d3db76ab857a79953ee25b6964538497a8448d487f2c"
Sep 07 13:11:20.077690 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:20.077672    2565 scope.go:117] "RemoveContainer" containerID="ff51de4b535c3da7dfabf0f2947853c3734588b06387e4423bd619fc2387dab3"
Sep 07 13:11:20.086038 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:20.085865    2565 scope.go:117] "RemoveContainer" containerID="e2fb0187527a4bf473d2d3db76ab857a79953ee25b6964538497a8448d487f2c"
Sep 07 13:11:20.086178 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:11:20.086158    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2fb0187527a4bf473d2d3db76ab857a79953ee25b6964538497a8448d487f2c\": container with ID starting with e2fb0187527a4bf473d2d3db76ab857a79953ee25b6964538497a8448d487f2c not found: ID does not exist" containerID="e2fb0187527a4bf473d2d3db76ab857a79953ee25b6964538497a8448d487f2c"
Sep 07 13:11:20.086224 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:20.086187    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2fb0187527a4bf473d2d3db76ab857a79953ee25b6964538497a8448d487f2c"} err="failed to get container status \"e2fb0187527a4bf473d2d3db76ab857a79953ee25b6964538497a8448d487f2c\": rpc error: code = NotFound desc = could not find container \"e2fb0187527a4bf473d2d3db76ab857a79953ee25b6964538497a8448d487f2c\": container with ID starting with e2fb0187527a4bf473d2d3db76ab857a79953ee25b6964538497a8448d487f2c not found: ID does not exist"
Sep 07 13:11:20.086224 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:20.086206    2565 scope.go:117] "RemoveContainer" containerID="ff51de4b535c3da7dfabf0f2947853c3734588b06387e4423bd619fc2387dab3"
Sep 07 13:11:20.086480 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:11:20.086460    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff51de4b535c3da7dfabf0f2947853c3734588b06387e4423bd619fc2387dab3\": container with ID starting with ff51de4b535c3da7dfabf0f2947853c3734588b06387e4423bd619fc2387dab3 not found: ID does not exist" containerID="ff51de4b535c3da7dfabf0f2947853c3734588b06387e4423bd619fc2387dab3"
Sep 07 13:11:20.086533 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:20.086488    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff51de4b535c3da7dfabf0f2947853c3734588b06387e4423bd619fc2387dab3"} err="failed to get container status \"ff51de4b535c3da7dfabf0f2947853c3734588b06387e4423bd619fc2387dab3\": rpc error: code = NotFound desc = could not find container \"ff51de4b535c3da7dfabf0f2947853c3734588b06387e4423bd619fc2387dab3\": container with ID starting with ff51de4b535c3da7dfabf0f2947853c3734588b06387e4423bd619fc2387dab3 not found: ID does not exist"
Sep 07 13:11:20.090951 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:20.090929    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"]
Sep 07 13:11:20.093055 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:20.093036    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"
Sep 07 13:11:20.095700 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:20.095679    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-76f028-openshift-default-6b46657dcd-hpk29"]
Sep 07 13:11:20.372788 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:20.372763    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"]
Sep 07 13:11:20.374930 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:11:20.374901    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e21fb80_e2d9_425e_b336_449536271a12.slice/crio-eebc5c9d728d5bd9463187d08716de3c0868ace79397bd323c5c636064ab8908 WatchSource:0}: Error finding container eebc5c9d728d5bd9463187d08716de3c0868ace79397bd323c5c636064ab8908: Status 404 returned error can't find the container with id eebc5c9d728d5bd9463187d08716de3c0868ace79397bd323c5c636064ab8908
Sep 07 13:11:21.076369 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:21.076314    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh" event={"ID":"6e21fb80-e2d9-425e-b336-449536271a12","Type":"ContainerStarted","Data":"5ce433d25d54d5d5a851e12ff68b484aba68a447e73016941cfc9e321c3dc347"}
Sep 07 13:11:21.076579 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:21.076383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh" event={"ID":"6e21fb80-e2d9-425e-b336-449536271a12","Type":"ContainerStarted","Data":"eebc5c9d728d5bd9463187d08716de3c0868ace79397bd323c5c636064ab8908"}
Sep 07 13:11:21.076579 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:21.076494    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"
Sep 07 13:11:21.103529 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:21.103481    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh" podStartSLOduration=2.103466971 podStartE2EDuration="2.103466971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:11:21.101797674 +0000 UTC m=+1098.132663999" watchObservedRunningTime="2026-09-07 13:11:21.103466971 +0000 UTC m=+1098.134333281"
Sep 07 13:11:21.488993 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:21.488912    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30ece672-58d2-489b-a652-9308ee1960d5" path="/var/lib/kubelet/pods/30ece672-58d2-489b-a652-9308ee1960d5/volumes"
Sep 07 13:11:21.918378 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:21.918343    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:11:32.085436 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:32.085404    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"
Sep 07 13:11:37.329370 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:37.329314    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k"]
Sep 07 13:11:37.329878 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:37.329803    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ece672-58d2-489b-a652-9308ee1960d5" containerName="istio-proxy"
Sep 07 13:11:37.333123 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:37.333102    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k"
Sep 07 13:11:37.335707 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:37.335684    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flngs\""
Sep 07 13:11:37.335830 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:37.335717    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 13:11:37.345499 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:37.345475    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k"]
Sep 07 13:11:37.486493 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:37.486457    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f348d4-18d0-43ac-8703-fce33aa508ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k\" (UID: \"b2f348d4-18d0-43ac-8703-fce33aa508ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k"
Sep 07 13:11:37.587615 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:37.587534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f348d4-18d0-43ac-8703-fce33aa508ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k\" (UID: \"b2f348d4-18d0-43ac-8703-fce33aa508ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k"
Sep 07 13:11:37.588161 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:37.588137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f348d4-18d0-43ac-8703-fce33aa508ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k\" (UID: \"b2f348d4-18d0-43ac-8703-fce33aa508ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k"
Sep 07 13:11:37.643927 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:37.643897    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k"
Sep 07 13:11:37.921775 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:37.921680    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k"]
Sep 07 13:11:37.924305 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:11:37.924274    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f348d4_18d0_43ac_8703_fce33aa508ab.slice/crio-2cbf4e39f76426ee80cddb2637860f906555c4c33a1c47d33e4613706629e510 WatchSource:0}: Error finding container 2cbf4e39f76426ee80cddb2637860f906555c4c33a1c47d33e4613706629e510: Status 404 returned error can't find the container with id 2cbf4e39f76426ee80cddb2637860f906555c4c33a1c47d33e4613706629e510
Sep 07 13:11:38.169735 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:38.169702    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k" event={"ID":"b2f348d4-18d0-43ac-8703-fce33aa508ab","Type":"ContainerStarted","Data":"2cbf4e39f76426ee80cddb2637860f906555c4c33a1c47d33e4613706629e510"}
Sep 07 13:11:39.177278 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:39.177240    2565 generic.go:356] "Generic (PLEG): container finished" podID="b2f348d4-18d0-43ac-8703-fce33aa508ab" containerID="6695d2b381ab7e0dc9f60c9fbea7f9e34614162569e780287925e2956a7b3491" exitCode=0
Sep 07 13:11:39.177664 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:39.177292    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k" event={"ID":"b2f348d4-18d0-43ac-8703-fce33aa508ab","Type":"ContainerDied","Data":"6695d2b381ab7e0dc9f60c9fbea7f9e34614162569e780287925e2956a7b3491"}
Sep 07 13:11:40.315742 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:40.315716    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k"
Sep 07 13:11:40.417427 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:40.417395    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f348d4-18d0-43ac-8703-fce33aa508ab-maas-api-service-ca\") pod \"b2f348d4-18d0-43ac-8703-fce33aa508ab\" (UID: \"b2f348d4-18d0-43ac-8703-fce33aa508ab\") "
Sep 07 13:11:40.417785 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:40.417756    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f348d4-18d0-43ac-8703-fce33aa508ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2f348d4-18d0-43ac-8703-fce33aa508ab" (UID: "b2f348d4-18d0-43ac-8703-fce33aa508ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:11:40.518651 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:40.518564    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f348d4-18d0-43ac-8703-fce33aa508ab-maas-api-service-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:41.189055 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:41.189023    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k"
Sep 07 13:11:41.189055 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:41.189040    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k" event={"ID":"b2f348d4-18d0-43ac-8703-fce33aa508ab","Type":"ContainerDied","Data":"2cbf4e39f76426ee80cddb2637860f906555c4c33a1c47d33e4613706629e510"}
Sep 07 13:11:41.189266 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:41.189082    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cbf4e39f76426ee80cddb2637860f906555c4c33a1c47d33e4613706629e510"
Sep 07 13:11:50.309527 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.309493    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"]
Sep 07 13:11:50.309916 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.309903    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f348d4-18d0-43ac-8703-fce33aa508ab" containerName="revoke-keys"
Sep 07 13:11:50.309977 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.309917    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f348d4-18d0-43ac-8703-fce33aa508ab" containerName="revoke-keys"
Sep 07 13:11:50.310014 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.309981    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f348d4-18d0-43ac-8703-fce33aa508ab" containerName="revoke-keys"
Sep 07 13:11:50.313268 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.313249    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"
Sep 07 13:11:50.314011 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.313985    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dxhb\" (UniqueName: \"kubernetes.io/projected/fc5708f7-7362-4455-9d1f-93336590f962-kube-api-access-5dxhb\") pod \"payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh\" (UID: \"fc5708f7-7362-4455-9d1f-93336590f962\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"
Sep 07 13:11:50.314124 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.314107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5708f7-7362-4455-9d1f-93336590f962-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh\" (UID: \"fc5708f7-7362-4455-9d1f-93336590f962\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"
Sep 07 13:11:50.315679 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.315663    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-ae93b0-dockercfg-2dqcw\""
Sep 07 13:11:50.316389 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.316370    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-ae93b0\""
Sep 07 13:11:50.328356 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.328297    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"]
Sep 07 13:11:50.414730 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.414692    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5708f7-7362-4455-9d1f-93336590f962-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh\" (UID: \"fc5708f7-7362-4455-9d1f-93336590f962\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"
Sep 07 13:11:50.414894 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.414772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dxhb\" (UniqueName: \"kubernetes.io/projected/fc5708f7-7362-4455-9d1f-93336590f962-kube-api-access-5dxhb\") pod \"payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh\" (UID: \"fc5708f7-7362-4455-9d1f-93336590f962\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"
Sep 07 13:11:50.415427 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.415408    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5708f7-7362-4455-9d1f-93336590f962-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh\" (UID: \"fc5708f7-7362-4455-9d1f-93336590f962\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"
Sep 07 13:11:50.432094 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.432065    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dxhb\" (UniqueName: \"kubernetes.io/projected/fc5708f7-7362-4455-9d1f-93336590f962-kube-api-access-5dxhb\") pod \"payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh\" (UID: \"fc5708f7-7362-4455-9d1f-93336590f962\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"
Sep 07 13:11:50.623764 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.623668    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"
Sep 07 13:11:50.730574 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.730537    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"]
Sep 07 13:11:50.730783 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.730753    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp" podUID="d7fa12ae-eeca-46ef-b0d3-a1e0b5126337" containerName="payload-processing" containerID="cri-o://bed2dc8d254af81f51e046b5d0e869a93c75229e9e6d25666e09c0deddbcbef3" gracePeriod=30
Sep 07 13:11:50.742682 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.742649    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"]
Sep 07 13:11:50.742966 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.742916    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4" podUID="f03b3760-9702-4d75-bedd-f337b7454f6f" containerName="payload-pre-processing" containerID="cri-o://531bccd0af5fb405450d46e735af843ac3c6d4012f028b48bbfe4e1377d120d9" gracePeriod=30
Sep 07 13:11:50.908273 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.908223    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"]
Sep 07 13:11:50.911028 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:11:50.910997    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5708f7_7362_4455_9d1f_93336590f962.slice/crio-132865436ba6df27a12e46a268751cf569e2b65d8822279d0d29c36df0c9445c WatchSource:0}: Error finding container 132865436ba6df27a12e46a268751cf569e2b65d8822279d0d29c36df0c9445c: Status 404 returned error can't find the container with id 132865436ba6df27a12e46a268751cf569e2b65d8822279d0d29c36df0c9445c
Sep 07 13:11:50.927285 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.927260    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k"]
Sep 07 13:11:50.934570 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:50.934548    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-953bf16f-lwv6k"]
Sep 07 13:11:51.005871 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.005847    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"
Sep 07 13:11:51.012483 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.012458    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"
Sep 07 13:11:51.018953 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.018923    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7fa12ae-eeca-46ef-b0d3-a1e0b5126337-plugins-config-volume\") pod \"d7fa12ae-eeca-46ef-b0d3-a1e0b5126337\" (UID: \"d7fa12ae-eeca-46ef-b0d3-a1e0b5126337\") "
Sep 07 13:11:51.019079 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.019047    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6mbb\" (UniqueName: \"kubernetes.io/projected/d7fa12ae-eeca-46ef-b0d3-a1e0b5126337-kube-api-access-z6mbb\") pod \"d7fa12ae-eeca-46ef-b0d3-a1e0b5126337\" (UID: \"d7fa12ae-eeca-46ef-b0d3-a1e0b5126337\") "
Sep 07 13:11:51.019145 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.019107    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptkz7\" (UniqueName: \"kubernetes.io/projected/f03b3760-9702-4d75-bedd-f337b7454f6f-kube-api-access-ptkz7\") pod \"f03b3760-9702-4d75-bedd-f337b7454f6f\" (UID: \"f03b3760-9702-4d75-bedd-f337b7454f6f\") "
Sep 07 13:11:51.019204 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.019157    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03b3760-9702-4d75-bedd-f337b7454f6f-plugins-config-volume\") pod \"f03b3760-9702-4d75-bedd-f337b7454f6f\" (UID: \"f03b3760-9702-4d75-bedd-f337b7454f6f\") "
Sep 07 13:11:51.019291 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.019261    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7fa12ae-eeca-46ef-b0d3-a1e0b5126337-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d7fa12ae-eeca-46ef-b0d3-a1e0b5126337" (UID: "d7fa12ae-eeca-46ef-b0d3-a1e0b5126337"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:11:51.019621 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.019603    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7fa12ae-eeca-46ef-b0d3-a1e0b5126337-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:51.019701 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.019625    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03b3760-9702-4d75-bedd-f337b7454f6f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f03b3760-9702-4d75-bedd-f337b7454f6f" (UID: "f03b3760-9702-4d75-bedd-f337b7454f6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:11:51.021422 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.021398    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7fa12ae-eeca-46ef-b0d3-a1e0b5126337-kube-api-access-z6mbb" (OuterVolumeSpecName: "kube-api-access-z6mbb") pod "d7fa12ae-eeca-46ef-b0d3-a1e0b5126337" (UID: "d7fa12ae-eeca-46ef-b0d3-a1e0b5126337"). InnerVolumeSpecName "kube-api-access-z6mbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:11:51.021581 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.021561    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03b3760-9702-4d75-bedd-f337b7454f6f-kube-api-access-ptkz7" (OuterVolumeSpecName: "kube-api-access-ptkz7") pod "f03b3760-9702-4d75-bedd-f337b7454f6f" (UID: "f03b3760-9702-4d75-bedd-f337b7454f6f"). InnerVolumeSpecName "kube-api-access-ptkz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:11:51.120756 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.120710    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6mbb\" (UniqueName: \"kubernetes.io/projected/d7fa12ae-eeca-46ef-b0d3-a1e0b5126337-kube-api-access-z6mbb\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:51.120917 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.120750    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptkz7\" (UniqueName: \"kubernetes.io/projected/f03b3760-9702-4d75-bedd-f337b7454f6f-kube-api-access-ptkz7\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:51.120917 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.120784    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03b3760-9702-4d75-bedd-f337b7454f6f-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:11:51.243215 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.243122    2565 generic.go:356] "Generic (PLEG): container finished" podID="d7fa12ae-eeca-46ef-b0d3-a1e0b5126337" containerID="bed2dc8d254af81f51e046b5d0e869a93c75229e9e6d25666e09c0deddbcbef3" exitCode=0
Sep 07 13:11:51.243215 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.243175    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"
Sep 07 13:11:51.243215 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.243199    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp" event={"ID":"d7fa12ae-eeca-46ef-b0d3-a1e0b5126337","Type":"ContainerDied","Data":"bed2dc8d254af81f51e046b5d0e869a93c75229e9e6d25666e09c0deddbcbef3"}
Sep 07 13:11:51.243541 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.243253    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp" event={"ID":"d7fa12ae-eeca-46ef-b0d3-a1e0b5126337","Type":"ContainerDied","Data":"91fb5f5197cf8c2eab4bd0fde9c26bd578932bd632602bc52a67a05c77ff741d"}
Sep 07 13:11:51.243541 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.243283    2565 scope.go:117] "RemoveContainer" containerID="bed2dc8d254af81f51e046b5d0e869a93c75229e9e6d25666e09c0deddbcbef3"
Sep 07 13:11:51.244784 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.244760    2565 generic.go:356] "Generic (PLEG): container finished" podID="f03b3760-9702-4d75-bedd-f337b7454f6f" containerID="531bccd0af5fb405450d46e735af843ac3c6d4012f028b48bbfe4e1377d120d9" exitCode=0
Sep 07 13:11:51.244887 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.244856    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4" event={"ID":"f03b3760-9702-4d75-bedd-f337b7454f6f","Type":"ContainerDied","Data":"531bccd0af5fb405450d46e735af843ac3c6d4012f028b48bbfe4e1377d120d9"}
Sep 07 13:11:51.244990 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.244919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4" event={"ID":"f03b3760-9702-4d75-bedd-f337b7454f6f","Type":"ContainerDied","Data":"2aec21abd7938b7686e6d1f52bdce48581b8d9d881dca6da8e4e4124774f1e37"}
Sep 07 13:11:51.244990 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.244982    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"
Sep 07 13:11:51.246634 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.246608    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh" event={"ID":"fc5708f7-7362-4455-9d1f-93336590f962","Type":"ContainerStarted","Data":"1d469f63b7f70d5c97abe38c6bd173ecbece40acb980fcacf5c0091c4ce904af"}
Sep 07 13:11:51.246770 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.246699    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"
Sep 07 13:11:51.246770 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.246729    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh" event={"ID":"fc5708f7-7362-4455-9d1f-93336590f962","Type":"ContainerStarted","Data":"132865436ba6df27a12e46a268751cf569e2b65d8822279d0d29c36df0c9445c"}
Sep 07 13:11:51.254454 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.254435    2565 scope.go:117] "RemoveContainer" containerID="bed2dc8d254af81f51e046b5d0e869a93c75229e9e6d25666e09c0deddbcbef3"
Sep 07 13:11:51.254766 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:11:51.254734    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed2dc8d254af81f51e046b5d0e869a93c75229e9e6d25666e09c0deddbcbef3\": container with ID starting with bed2dc8d254af81f51e046b5d0e869a93c75229e9e6d25666e09c0deddbcbef3 not found: ID does not exist" containerID="bed2dc8d254af81f51e046b5d0e869a93c75229e9e6d25666e09c0deddbcbef3"
Sep 07 13:11:51.254887 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.254776    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed2dc8d254af81f51e046b5d0e869a93c75229e9e6d25666e09c0deddbcbef3"} err="failed to get container status \"bed2dc8d254af81f51e046b5d0e869a93c75229e9e6d25666e09c0deddbcbef3\": rpc error: code = NotFound desc = could not find container \"bed2dc8d254af81f51e046b5d0e869a93c75229e9e6d25666e09c0deddbcbef3\": container with ID starting with bed2dc8d254af81f51e046b5d0e869a93c75229e9e6d25666e09c0deddbcbef3 not found: ID does not exist"
Sep 07 13:11:51.254887 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.254795    2565 scope.go:117] "RemoveContainer" containerID="531bccd0af5fb405450d46e735af843ac3c6d4012f028b48bbfe4e1377d120d9"
Sep 07 13:11:51.262823 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.262806    2565 scope.go:117] "RemoveContainer" containerID="531bccd0af5fb405450d46e735af843ac3c6d4012f028b48bbfe4e1377d120d9"
Sep 07 13:11:51.263070 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:11:51.263051    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531bccd0af5fb405450d46e735af843ac3c6d4012f028b48bbfe4e1377d120d9\": container with ID starting with 531bccd0af5fb405450d46e735af843ac3c6d4012f028b48bbfe4e1377d120d9 not found: ID does not exist" containerID="531bccd0af5fb405450d46e735af843ac3c6d4012f028b48bbfe4e1377d120d9"
Sep 07 13:11:51.263131 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.263080    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531bccd0af5fb405450d46e735af843ac3c6d4012f028b48bbfe4e1377d120d9"} err="failed to get container status \"531bccd0af5fb405450d46e735af843ac3c6d4012f028b48bbfe4e1377d120d9\": rpc error: code = NotFound desc = could not find container \"531bccd0af5fb405450d46e735af843ac3c6d4012f028b48bbfe4e1377d120d9\": container with ID starting with 531bccd0af5fb405450d46e735af843ac3c6d4012f028b48bbfe4e1377d120d9 not found: ID does not exist"
Sep 07 13:11:51.274787 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.274744    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh" podStartSLOduration=1.274730827 podStartE2EDuration="1.274730827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:11:51.273031305 +0000 UTC m=+1128.303897615" watchObservedRunningTime="2026-09-07 13:11:51.274730827 +0000 UTC m=+1128.305597136"
Sep 07 13:11:51.292411 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.292388    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"]
Sep 07 13:11:51.297597 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.297577    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-953bf16f-7d569b5d58-csrf4"]
Sep 07 13:11:51.311988 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.311968    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"]
Sep 07 13:11:51.318653 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.318634    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-953bf16f-554749d78f-zk7cp"]
Sep 07 13:11:51.487919 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.487885    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2f348d4-18d0-43ac-8703-fce33aa508ab" path="/var/lib/kubelet/pods/b2f348d4-18d0-43ac-8703-fce33aa508ab/volumes"
Sep 07 13:11:51.488251 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.488238    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7fa12ae-eeca-46ef-b0d3-a1e0b5126337" path="/var/lib/kubelet/pods/d7fa12ae-eeca-46ef-b0d3-a1e0b5126337/volumes"
Sep 07 13:11:51.488586 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:51.488574    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f03b3760-9702-4d75-bedd-f337b7454f6f" path="/var/lib/kubelet/pods/f03b3760-9702-4d75-bedd-f337b7454f6f/volumes"
Sep 07 13:11:53.416296 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:11:53.416258    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:12:02.256054 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:02.256019    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"
Sep 07 13:12:15.010975 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.010944    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"]
Sep 07 13:12:15.011440 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.011424    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03b3760-9702-4d75-bedd-f337b7454f6f" containerName="payload-pre-processing"
Sep 07 13:12:15.011492 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.011443    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03b3760-9702-4d75-bedd-f337b7454f6f" containerName="payload-pre-processing"
Sep 07 13:12:15.011492 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.011458    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7fa12ae-eeca-46ef-b0d3-a1e0b5126337" containerName="payload-processing"
Sep 07 13:12:15.011492 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.011463    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7fa12ae-eeca-46ef-b0d3-a1e0b5126337" containerName="payload-processing"
Sep 07 13:12:15.011607 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.011532    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7fa12ae-eeca-46ef-b0d3-a1e0b5126337" containerName="payload-processing"
Sep 07 13:12:15.011607 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.011546    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03b3760-9702-4d75-bedd-f337b7454f6f" containerName="payload-pre-processing"
Sep 07 13:12:15.016014 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.015994    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.018781 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.018757    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-bc4b05-openshift-default-dockercfg-tmt29\""
Sep 07 13:12:15.030625 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.030602    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"]
Sep 07 13:12:15.162112 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.162075    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45828b52-0938-4d26-942c-28fe56c6c212-istiod-ca-cert\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.162112 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.162116    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-istio-data\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.162433 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.162196    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gqjw\" (UniqueName: \"kubernetes.io/projected/45828b52-0938-4d26-942c-28fe56c6c212-kube-api-access-7gqjw\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.162433 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.162262    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45828b52-0938-4d26-942c-28fe56c6c212-istio-podinfo\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.162433 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.162409    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-workload-certs\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.162600 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.162452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-istio-envoy\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.162600 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.162504    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-credential-socket\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.162600 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.162549    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-workload-socket\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.162600 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.162595    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45828b52-0938-4d26-942c-28fe56c6c212-istio-token\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.263740 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.263650    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-workload-socket\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.263740 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.263690    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45828b52-0938-4d26-942c-28fe56c6c212-istio-token\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.263740 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.263719    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45828b52-0938-4d26-942c-28fe56c6c212-istiod-ca-cert\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.263740 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.263743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-istio-data\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.264075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.263763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gqjw\" (UniqueName: \"kubernetes.io/projected/45828b52-0938-4d26-942c-28fe56c6c212-kube-api-access-7gqjw\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.264075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.263788    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45828b52-0938-4d26-942c-28fe56c6c212-istio-podinfo\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.264075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.263850    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-workload-certs\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.264075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.263887    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-istio-envoy\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.264075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.263916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-credential-socket\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.264366 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.264122    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-workload-socket\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.264366 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.264263    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-istio-data\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.264366 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.264289    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-credential-socket\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.264366 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.264312    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-workload-certs\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.264601 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.264578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45828b52-0938-4d26-942c-28fe56c6c212-istiod-ca-cert\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.266481 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.266458    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-istio-envoy\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.266671 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.266649    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45828b52-0938-4d26-942c-28fe56c6c212-istio-podinfo\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.274101 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.274074    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gqjw\" (UniqueName: \"kubernetes.io/projected/45828b52-0938-4d26-942c-28fe56c6c212-kube-api-access-7gqjw\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.274464 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.274441    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45828b52-0938-4d26-942c-28fe56c6c212-istio-token\") pod \"e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") " pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.326711 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.326676    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:15.606591 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.606559    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"]
Sep 07 13:12:15.610395 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:12:15.610358    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45828b52_0938_4d26_942c_28fe56c6c212.slice/crio-fcb89e96dc8b8fd57061147fb018f75962a7ae4e5eb92bafdd15732ae9d36359 WatchSource:0}: Error finding container fcb89e96dc8b8fd57061147fb018f75962a7ae4e5eb92bafdd15732ae9d36359: Status 404 returned error can't find the container with id fcb89e96dc8b8fd57061147fb018f75962a7ae4e5eb92bafdd15732ae9d36359
Sep 07 13:12:15.613081 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.613050    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:12:15.613159 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.613119    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:12:15.613159 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:15.613150    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:12:16.388037 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:16.387997    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd" event={"ID":"45828b52-0938-4d26-942c-28fe56c6c212","Type":"ContainerStarted","Data":"0fd34a4d783cd2240ca4cc65568dc9a504661920cd48d12148af90fdd069531f"}
Sep 07 13:12:16.388494 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:16.388050    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd" event={"ID":"45828b52-0938-4d26-942c-28fe56c6c212","Type":"ContainerStarted","Data":"fcb89e96dc8b8fd57061147fb018f75962a7ae4e5eb92bafdd15732ae9d36359"}
Sep 07 13:12:16.415646 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:16.415596    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd" podStartSLOduration=2.415580158 podStartE2EDuration="2.415580158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:12:16.413136062 +0000 UTC m=+1153.444002377" watchObservedRunningTime="2026-09-07 13:12:16.415580158 +0000 UTC m=+1153.446446467"
Sep 07 13:12:17.327878 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:17.327839    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:17.332784 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:17.332760    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:17.393442 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:17.393401    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:17.394318 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:17.394298    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:20.256584 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.256547    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"]
Sep 07 13:12:20.260280 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.260261    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"
Sep 07 13:12:20.262839 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.262811    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-bc4b05-dockercfg-pzcnv\""
Sep 07 13:12:20.262981 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.262838    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-bc4b05\""
Sep 07 13:12:20.271824 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.271800    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"]
Sep 07 13:12:20.301103 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.301064    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"]
Sep 07 13:12:20.304863 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.304843    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"
Sep 07 13:12:20.316277 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.316251    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"]
Sep 07 13:12:20.417350 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.417297    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts22g\" (UniqueName: \"kubernetes.io/projected/4aeafa44-ad23-41a6-8ce8-6d42045202d5-kube-api-access-ts22g\") pod \"payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5\" (UID: \"4aeafa44-ad23-41a6-8ce8-6d42045202d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"
Sep 07 13:12:20.417518 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.417374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aeafa44-ad23-41a6-8ce8-6d42045202d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5\" (UID: \"4aeafa44-ad23-41a6-8ce8-6d42045202d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"
Sep 07 13:12:20.417518 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.417433    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-869xx\" (UniqueName: \"kubernetes.io/projected/fcc77009-2064-4063-8512-b913d71f3d4b-kube-api-access-869xx\") pod \"payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g\" (UID: \"fcc77009-2064-4063-8512-b913d71f3d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"
Sep 07 13:12:20.417518 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.417496    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc77009-2064-4063-8512-b913d71f3d4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g\" (UID: \"fcc77009-2064-4063-8512-b913d71f3d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"
Sep 07 13:12:20.518699 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.518607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-869xx\" (UniqueName: \"kubernetes.io/projected/fcc77009-2064-4063-8512-b913d71f3d4b-kube-api-access-869xx\") pod \"payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g\" (UID: \"fcc77009-2064-4063-8512-b913d71f3d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"
Sep 07 13:12:20.518859 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.518729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc77009-2064-4063-8512-b913d71f3d4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g\" (UID: \"fcc77009-2064-4063-8512-b913d71f3d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"
Sep 07 13:12:20.518859 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.518778    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts22g\" (UniqueName: \"kubernetes.io/projected/4aeafa44-ad23-41a6-8ce8-6d42045202d5-kube-api-access-ts22g\") pod \"payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5\" (UID: \"4aeafa44-ad23-41a6-8ce8-6d42045202d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"
Sep 07 13:12:20.518859 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.518813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aeafa44-ad23-41a6-8ce8-6d42045202d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5\" (UID: \"4aeafa44-ad23-41a6-8ce8-6d42045202d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"
Sep 07 13:12:20.519672 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.519649    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc77009-2064-4063-8512-b913d71f3d4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g\" (UID: \"fcc77009-2064-4063-8512-b913d71f3d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"
Sep 07 13:12:20.519721 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.519649    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aeafa44-ad23-41a6-8ce8-6d42045202d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5\" (UID: \"4aeafa44-ad23-41a6-8ce8-6d42045202d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"
Sep 07 13:12:20.531782 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.531750    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-869xx\" (UniqueName: \"kubernetes.io/projected/fcc77009-2064-4063-8512-b913d71f3d4b-kube-api-access-869xx\") pod \"payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g\" (UID: \"fcc77009-2064-4063-8512-b913d71f3d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"
Sep 07 13:12:20.531930 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.531790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts22g\" (UniqueName: \"kubernetes.io/projected/4aeafa44-ad23-41a6-8ce8-6d42045202d5-kube-api-access-ts22g\") pod \"payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5\" (UID: \"4aeafa44-ad23-41a6-8ce8-6d42045202d5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"
Sep 07 13:12:20.571660 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.571633    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"
Sep 07 13:12:20.616346 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.616299    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"
Sep 07 13:12:20.880617 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.880468    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"]
Sep 07 13:12:20.883002 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:12:20.882973    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aeafa44_ad23_41a6_8ce8_6d42045202d5.slice/crio-1affdcb33c22f8ab84787ecbe5e23521223fbdb80fa258ca0ed961f0ab0ffb08 WatchSource:0}: Error finding container 1affdcb33c22f8ab84787ecbe5e23521223fbdb80fa258ca0ed961f0ab0ffb08: Status 404 returned error can't find the container with id 1affdcb33c22f8ab84787ecbe5e23521223fbdb80fa258ca0ed961f0ab0ffb08
Sep 07 13:12:20.924112 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:20.924084    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"]
Sep 07 13:12:20.927023 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:12:20.926999    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc77009_2064_4063_8512_b913d71f3d4b.slice/crio-04fc2fe990c505ee76b5c3a5e1dc9ee10fbf6e6d273f17a568f7db7c8f479881 WatchSource:0}: Error finding container 04fc2fe990c505ee76b5c3a5e1dc9ee10fbf6e6d273f17a568f7db7c8f479881: Status 404 returned error can't find the container with id 04fc2fe990c505ee76b5c3a5e1dc9ee10fbf6e6d273f17a568f7db7c8f479881
Sep 07 13:12:21.415492 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:21.415451    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5" event={"ID":"4aeafa44-ad23-41a6-8ce8-6d42045202d5","Type":"ContainerStarted","Data":"17bca8a9ec70af7f57ddbb6247952583196ff1c3fd193b2026a35df862cfe3bc"}
Sep 07 13:12:21.415986 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:21.415518    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5" event={"ID":"4aeafa44-ad23-41a6-8ce8-6d42045202d5","Type":"ContainerStarted","Data":"1affdcb33c22f8ab84787ecbe5e23521223fbdb80fa258ca0ed961f0ab0ffb08"}
Sep 07 13:12:21.415986 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:21.415547    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"
Sep 07 13:12:21.417031 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:21.417000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g" event={"ID":"fcc77009-2064-4063-8512-b913d71f3d4b","Type":"ContainerStarted","Data":"34dcc8a1cdacdde2ac039b7a09bde4612c28ad0e54b6151a6fe08a924cfc5ee3"}
Sep 07 13:12:21.417130 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:21.417053    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"
Sep 07 13:12:21.417130 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:21.417072    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g" event={"ID":"fcc77009-2064-4063-8512-b913d71f3d4b","Type":"ContainerStarted","Data":"04fc2fe990c505ee76b5c3a5e1dc9ee10fbf6e6d273f17a568f7db7c8f479881"}
Sep 07 13:12:21.437686 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:21.437631    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5" podStartSLOduration=1.437615383 podStartE2EDuration="1.437615383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:12:21.434036687 +0000 UTC m=+1158.464902998" watchObservedRunningTime="2026-09-07 13:12:21.437615383 +0000 UTC m=+1158.468481693"
Sep 07 13:12:21.453694 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:21.453647    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g" podStartSLOduration=1.453630586 podStartE2EDuration="1.453630586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:12:21.451949446 +0000 UTC m=+1158.482815759" watchObservedRunningTime="2026-09-07 13:12:21.453630586 +0000 UTC m=+1158.484496896"
Sep 07 13:12:25.207400 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:25.207287    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:12:29.214179 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.214143    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"]
Sep 07 13:12:29.219083 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.219065    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.222315 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.222292    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-aa813e\"/\"auth-test-model-aa813e-kserve-self-signed-certs\""
Sep 07 13:12:29.222902 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.222883    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-aa813e\"/\"kube-root-ca.crt\""
Sep 07 13:12:29.223064 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.223047    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-aa813e\"/\"default-dockercfg-nlkfv\""
Sep 07 13:12:29.223138 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.223086    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-aa813e\"/\"openshift-service-ca.crt\""
Sep 07 13:12:29.228707 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.228684    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"]
Sep 07 13:12:29.309045 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.307903    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-tmp-dir\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.309045 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.308004    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-home\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.309045 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.308034    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fdgd\" (UniqueName: \"kubernetes.io/projected/028abb1f-67e9-43bc-9c65-44456ab0fedf-kube-api-access-7fdgd\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.309045 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.308128    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-model-cache\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.309045 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.308154    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028abb1f-67e9-43bc-9c65-44456ab0fedf-tls-certs\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.309045 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.308202    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-dshm\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.408994 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.408959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-tmp-dir\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.408994 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.409003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-home\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.409261 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.409019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fdgd\" (UniqueName: \"kubernetes.io/projected/028abb1f-67e9-43bc-9c65-44456ab0fedf-kube-api-access-7fdgd\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.409261 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.409159    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-model-cache\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.409261 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.409202    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028abb1f-67e9-43bc-9c65-44456ab0fedf-tls-certs\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.409261 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.409257    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-dshm\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.409495 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.409433    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-tmp-dir\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.409541 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.409510    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-home\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.409541 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.409526    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-model-cache\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.411603 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.411574    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-dshm\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.411882 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.411864    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028abb1f-67e9-43bc-9c65-44456ab0fedf-tls-certs\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.420466 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.420445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fdgd\" (UniqueName: \"kubernetes.io/projected/028abb1f-67e9-43bc-9c65-44456ab0fedf-kube-api-access-7fdgd\") pod \"auth-test-model-aa813e-kserve-579cc9b687-5h97q\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.529624 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.529533    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:29.820788 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:29.820759    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"]
Sep 07 13:12:29.823157 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:12:29.823130    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod028abb1f_67e9_43bc_9c65_44456ab0fedf.slice/crio-abfb2feec3545b256142b97bf0fcc518039d778ec8e2f0e2fc5afe2a0deaf49d WatchSource:0}: Error finding container abfb2feec3545b256142b97bf0fcc518039d778ec8e2f0e2fc5afe2a0deaf49d: Status 404 returned error can't find the container with id abfb2feec3545b256142b97bf0fcc518039d778ec8e2f0e2fc5afe2a0deaf49d
Sep 07 13:12:30.466313 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:30.466271    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q" event={"ID":"028abb1f-67e9-43bc-9c65-44456ab0fedf","Type":"ContainerStarted","Data":"ec0cd26f1446beb577de9c8780913116a6a426c3bf7dde43666ff9dd6a664065"}
Sep 07 13:12:30.466774 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:30.466341    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q" event={"ID":"028abb1f-67e9-43bc-9c65-44456ab0fedf","Type":"ContainerStarted","Data":"abfb2feec3545b256142b97bf0fcc518039d778ec8e2f0e2fc5afe2a0deaf49d"}
Sep 07 13:12:30.499826 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:30.499761    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q" podStartSLOduration=1.499737089 podStartE2EDuration="1.499737089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:12:30.497243542 +0000 UTC m=+1167.528109853" watchObservedRunningTime="2026-09-07 13:12:30.499737089 +0000 UTC m=+1167.530603397"
Sep 07 13:12:32.425542 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:32.425504    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"
Sep 07 13:12:32.426070 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:32.425586    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"
Sep 07 13:12:33.435499 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:33.435458    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 13:12:33.435899 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:33.435553    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd" podUID="45828b52-0938-4d26-942c-28fe56c6c212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 13:12:34.311293 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:34.311254    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:12:35.495370 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:35.495317    2565 generic.go:356] "Generic (PLEG): container finished" podID="45828b52-0938-4d26-942c-28fe56c6c212" containerID="0fd34a4d783cd2240ca4cc65568dc9a504661920cd48d12148af90fdd069531f" exitCode=0
Sep 07 13:12:35.495788 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:35.495391    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd" event={"ID":"45828b52-0938-4d26-942c-28fe56c6c212","Type":"ContainerDied","Data":"0fd34a4d783cd2240ca4cc65568dc9a504661920cd48d12148af90fdd069531f"}
Sep 07 13:12:35.495832 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:35.495805    2565 scope.go:117] "RemoveContainer" containerID="0fd34a4d783cd2240ca4cc65568dc9a504661920cd48d12148af90fdd069531f"
Sep 07 13:12:35.496995 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:35.496967    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:12:35.497062 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:35.497036    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:12:35.497101 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:35.497067    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:12:36.506307 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:36.506270    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd" event={"ID":"45828b52-0938-4d26-942c-28fe56c6c212","Type":"ContainerStarted","Data":"745455cac6a8b91968c306718ac040ae6e30cc88b5b7ffd0b3b64e18e44d7dda"}
Sep 07 13:12:37.327411 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:37.327373    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:37.327614 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:37.327590    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 07 13:12:37.327662 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:37.327640    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd" podUID="45828b52-0938-4d26-942c-28fe56c6c212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 07 13:12:38.327755 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:38.327705    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 07 13:12:38.328253 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:38.327793    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd" podUID="45828b52-0938-4d26-942c-28fe56c6c212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 07 13:12:39.327720 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:39.327679    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 07 13:12:39.327915 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:39.327738    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd" podUID="45828b52-0938-4d26-942c-28fe56c6c212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 07 13:12:39.529778 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:39.529745    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:39.529950 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:39.529817    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:39.542362 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:39.542316    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:40.327505 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:40.327463    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 07 13:12:40.327731 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:40.327522    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd" podUID="45828b52-0938-4d26-942c-28fe56c6c212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 07 13:12:40.545965 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:40.545927    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:12:41.327927 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:41.327881    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 07 13:12:41.328105 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:41.327951    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd" podUID="45828b52-0938-4d26-942c-28fe56c6c212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 07 13:12:42.331461 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:42.331427    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:42.331851 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:42.331771    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:42.332542 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:42.332526    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:12:49.111383 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:49.111342    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"]
Sep 07 13:12:49.117700 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:49.116668    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"
Sep 07 13:12:49.119532 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:49.119507    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-45d721\""
Sep 07 13:12:49.119709 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:49.119507    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-45d721-dockercfg-dk6vw\""
Sep 07 13:12:49.127139 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:49.127115    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"]
Sep 07 13:12:49.220521 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:49.220477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4w4p\" (UniqueName: \"kubernetes.io/projected/32e78db2-72cb-498e-8375-50e4b30de76e-kube-api-access-w4w4p\") pod \"payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj\" (UID: \"32e78db2-72cb-498e-8375-50e4b30de76e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"
Sep 07 13:12:49.220707 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:49.220638    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32e78db2-72cb-498e-8375-50e4b30de76e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj\" (UID: \"32e78db2-72cb-498e-8375-50e4b30de76e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"
Sep 07 13:12:49.321792 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:49.321754    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32e78db2-72cb-498e-8375-50e4b30de76e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj\" (UID: \"32e78db2-72cb-498e-8375-50e4b30de76e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"
Sep 07 13:12:49.321986 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:49.321855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4w4p\" (UniqueName: \"kubernetes.io/projected/32e78db2-72cb-498e-8375-50e4b30de76e-kube-api-access-w4w4p\") pod \"payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj\" (UID: \"32e78db2-72cb-498e-8375-50e4b30de76e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"
Sep 07 13:12:49.322493 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:49.322470    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32e78db2-72cb-498e-8375-50e4b30de76e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj\" (UID: \"32e78db2-72cb-498e-8375-50e4b30de76e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"
Sep 07 13:12:49.330822 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:49.330800    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4w4p\" (UniqueName: \"kubernetes.io/projected/32e78db2-72cb-498e-8375-50e4b30de76e-kube-api-access-w4w4p\") pod \"payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj\" (UID: \"32e78db2-72cb-498e-8375-50e4b30de76e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"
Sep 07 13:12:49.432733 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:49.432631    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"
Sep 07 13:12:49.707852 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:49.707822    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"]
Sep 07 13:12:49.709704 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:12:49.709673    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e78db2_72cb_498e_8375_50e4b30de76e.slice/crio-aee9cee2c741cbd412029a6a0beb27f87e0ccce7dd86379894fa0b9970180b82 WatchSource:0}: Error finding container aee9cee2c741cbd412029a6a0beb27f87e0ccce7dd86379894fa0b9970180b82: Status 404 returned error can't find the container with id aee9cee2c741cbd412029a6a0beb27f87e0ccce7dd86379894fa0b9970180b82
Sep 07 13:12:50.595067 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:50.595015    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj" event={"ID":"32e78db2-72cb-498e-8375-50e4b30de76e","Type":"ContainerStarted","Data":"1e7eb689326c6902816625e69db105a2b8c2654f8d97d9e2413c73e14f865185"}
Sep 07 13:12:50.595555 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:50.595096    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj" event={"ID":"32e78db2-72cb-498e-8375-50e4b30de76e","Type":"ContainerStarted","Data":"aee9cee2c741cbd412029a6a0beb27f87e0ccce7dd86379894fa0b9970180b82"}
Sep 07 13:12:50.595555 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:50.595128    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"
Sep 07 13:12:50.615142 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:50.615084    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj" podStartSLOduration=1.6150625330000001 podStartE2EDuration="1.615062533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:12:50.613605246 +0000 UTC m=+1187.644471571" watchObservedRunningTime="2026-09-07 13:12:50.615062533 +0000 UTC m=+1187.645928844"
Sep 07 13:12:55.313373 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:12:55.313308    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:13:01.603433 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:01.603399    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"
Sep 07 13:13:12.113467 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:12.113429    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:13:25.255062 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.255027    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"]
Sep 07 13:13:25.261869 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.261832    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.264704 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.264672    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0561b5ad-openshift-default-dockercfg-wplxf\""
Sep 07 13:13:25.271482 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.271456    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"]
Sep 07 13:13:25.394540 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.394498    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad31d994-184e-4491-8f55-8f0208e5a127-istio-token\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.394735 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.394550    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-istio-envoy\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.394735 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.394613    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-workload-certs\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.394735 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.394664    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvqpr\" (UniqueName: \"kubernetes.io/projected/ad31d994-184e-4491-8f55-8f0208e5a127-kube-api-access-zvqpr\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.394735 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.394724    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad31d994-184e-4491-8f55-8f0208e5a127-istio-podinfo\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.394900 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.394747    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-credential-socket\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.394900 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.394790    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad31d994-184e-4491-8f55-8f0208e5a127-istiod-ca-cert\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.394900 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.394829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-workload-socket\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.394900 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.394858    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-istio-data\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.495648 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.495619    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvqpr\" (UniqueName: \"kubernetes.io/projected/ad31d994-184e-4491-8f55-8f0208e5a127-kube-api-access-zvqpr\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.495811 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.495694    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad31d994-184e-4491-8f55-8f0208e5a127-istio-podinfo\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.495811 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.495713    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-credential-socket\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.495811 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.495751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad31d994-184e-4491-8f55-8f0208e5a127-istiod-ca-cert\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.495811 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.495770    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-workload-socket\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.495811 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.495791    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-istio-data\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.496106 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.495836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad31d994-184e-4491-8f55-8f0208e5a127-istio-token\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.496106 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.495860    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-istio-envoy\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.496106 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.495884    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-workload-certs\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.496262 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.496214    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-credential-socket\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.496262 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.496216    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-workload-certs\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.496419 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.496263    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-istio-data\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.496459 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.496432    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-workload-socket\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.496621 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.496602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad31d994-184e-4491-8f55-8f0208e5a127-istiod-ca-cert\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.498768 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.498752    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-istio-envoy\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.499084 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.499063    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad31d994-184e-4491-8f55-8f0208e5a127-istio-podinfo\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.505103 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.505044    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad31d994-184e-4491-8f55-8f0208e5a127-istio-token\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.505203 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.505105    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvqpr\" (UniqueName: \"kubernetes.io/projected/ad31d994-184e-4491-8f55-8f0208e5a127-kube-api-access-zvqpr\") pod \"e2e-mt-0561b5ad-openshift-default-665649f664-w668g\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") " pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.574160 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.574122    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:25.857149 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.857122    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"]
Sep 07 13:13:25.859288 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:13:25.859258    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad31d994_184e_4491_8f55_8f0208e5a127.slice/crio-f3a7b2e3df3b243b6817f9f5dcb63cec30aea17dc6c54097e05170adc93fc68b WatchSource:0}: Error finding container f3a7b2e3df3b243b6817f9f5dcb63cec30aea17dc6c54097e05170adc93fc68b: Status 404 returned error can't find the container with id f3a7b2e3df3b243b6817f9f5dcb63cec30aea17dc6c54097e05170adc93fc68b
Sep 07 13:13:25.861815 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.861779    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:13:25.861909 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.861848    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:13:25.861909 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:25.861881    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:13:26.796008 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:26.795960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g" event={"ID":"ad31d994-184e-4491-8f55-8f0208e5a127","Type":"ContainerStarted","Data":"cfa6b9e6ed0c3a18b8461dcee3c2683c6153d5097ba6c4afd3e09ae59d7c954e"}
Sep 07 13:13:26.796496 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:26.796025    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g" event={"ID":"ad31d994-184e-4491-8f55-8f0208e5a127","Type":"ContainerStarted","Data":"f3a7b2e3df3b243b6817f9f5dcb63cec30aea17dc6c54097e05170adc93fc68b"}
Sep 07 13:13:26.821945 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:26.821882    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g" podStartSLOduration=1.8218590190000001 podStartE2EDuration="1.821859019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:13:26.82039568 +0000 UTC m=+1223.851261991" watchObservedRunningTime="2026-09-07 13:13:26.821859019 +0000 UTC m=+1223.852725330"
Sep 07 13:13:27.575022 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:27.574983    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:27.580662 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:27.580634    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:27.801574 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:27.801545    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:27.802701 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:27.802683    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:29.508910 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:29.508872    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:13:46.417069 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:46.417028    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:13:50.950857 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:50.950817    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad31d994-184e-4491-8f55-8f0208e5a127" containerID="cfa6b9e6ed0c3a18b8461dcee3c2683c6153d5097ba6c4afd3e09ae59d7c954e" exitCode=0
Sep 07 13:13:50.951284 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:50.950891    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g" event={"ID":"ad31d994-184e-4491-8f55-8f0208e5a127","Type":"ContainerDied","Data":"cfa6b9e6ed0c3a18b8461dcee3c2683c6153d5097ba6c4afd3e09ae59d7c954e"}
Sep 07 13:13:50.951389 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:50.951375    2565 scope.go:117] "RemoveContainer" containerID="cfa6b9e6ed0c3a18b8461dcee3c2683c6153d5097ba6c4afd3e09ae59d7c954e"
Sep 07 13:13:50.952525 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:50.952499    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:13:50.952592 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:50.952578    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:13:50.952628 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:50.952613    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:13:51.958646 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:51.958614    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g" event={"ID":"ad31d994-184e-4491-8f55-8f0208e5a127","Type":"ContainerStarted","Data":"79cec92a46b8c1c05cdfee7a280b5a5060adb1cfa336f1c76e1d553c711ac54a"}
Sep 07 13:13:52.574838 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:52.574796    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:52.575041 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:52.575018    2565 patch_prober.go:28] interesting pod/e2e-mt-0561b5ad-openshift-default-665649f664-w668g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 07 13:13:52.575104 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:52.575086    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g" podUID="ad31d994-184e-4491-8f55-8f0208e5a127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 07 13:13:53.575141 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:53.575105    2565 patch_prober.go:28] interesting pod/e2e-mt-0561b5ad-openshift-default-665649f664-w668g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 07 13:13:53.575542 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:53.575157    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g" podUID="ad31d994-184e-4491-8f55-8f0208e5a127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 07 13:13:54.575586 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:54.575546    2565 patch_prober.go:28] interesting pod/e2e-mt-0561b5ad-openshift-default-665649f664-w668g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 07 13:13:54.575984 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:54.575620    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g" podUID="ad31d994-184e-4491-8f55-8f0208e5a127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 07 13:13:55.575455 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:55.575414    2565 patch_prober.go:28] interesting pod/e2e-mt-0561b5ad-openshift-default-665649f664-w668g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 07 13:13:55.575680 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:55.575491    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g" podUID="ad31d994-184e-4491-8f55-8f0208e5a127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 07 13:13:56.575295 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:56.575256    2565 patch_prober.go:28] interesting pod/e2e-mt-0561b5ad-openshift-default-665649f664-w668g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 07 13:13:56.575493 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:56.575312    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g" podUID="ad31d994-184e-4491-8f55-8f0208e5a127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 07 13:13:57.579031 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:57.579000    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:57.579509 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:57.579280    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:13:57.580072 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:13:57.580056    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:14:02.006951 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.006912    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"]
Sep 07 13:14:02.011219 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.011198    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.013853 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.013831    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0561b5ad\"/\"openshift-service-ca.crt\""
Sep 07 13:14:02.013998 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.013975    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0561b5ad\"/\"kube-root-ca.crt\""
Sep 07 13:14:02.014075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.014027    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0561b5ad\"/\"default-dockercfg-gspd6\""
Sep 07 13:14:02.014848 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.014827    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0561b5ad\"/\"e2e-lifecycle-model-0561b5ad-kserve-self-signed-certs\""
Sep 07 13:14:02.022622 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.022596    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"]
Sep 07 13:14:02.089354 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.089295    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-home\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.089526 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.089417    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-tmp-dir\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.089526 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.089477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzlds\" (UniqueName: \"kubernetes.io/projected/9c609e16-b318-4b80-8113-15a6ebd303ba-kube-api-access-vzlds\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.089526 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.089522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-dshm\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.089632 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.089545    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c609e16-b318-4b80-8113-15a6ebd303ba-tls-certs\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.089632 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.089587    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-model-cache\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.190444 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.190404    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzlds\" (UniqueName: \"kubernetes.io/projected/9c609e16-b318-4b80-8113-15a6ebd303ba-kube-api-access-vzlds\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.190646 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.190482    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-dshm\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.190646 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.190519    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c609e16-b318-4b80-8113-15a6ebd303ba-tls-certs\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.190646 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.190543    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-model-cache\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.190646 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.190580    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-home\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.190874 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.190657    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-tmp-dir\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.191041 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.191018    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-model-cache\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.191109 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.191067    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-home\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.191109 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.191079    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-tmp-dir\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.192943 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.192921    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-dshm\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.193194 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.193176    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c609e16-b318-4b80-8113-15a6ebd303ba-tls-certs\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.201114 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.201087    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzlds\" (UniqueName: \"kubernetes.io/projected/9c609e16-b318-4b80-8113-15a6ebd303ba-kube-api-access-vzlds\") pod \"e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") " pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.322639 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.322601    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:02.599650 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:02.599620    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"]
Sep 07 13:14:02.601883 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:14:02.601858    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c609e16_b318_4b80_8113_15a6ebd303ba.slice/crio-75cfbe0d95a14522f3ca27cdb0beb1709896272a35ebeeca09ed9381a9c7d0a2 WatchSource:0}: Error finding container 75cfbe0d95a14522f3ca27cdb0beb1709896272a35ebeeca09ed9381a9c7d0a2: Status 404 returned error can't find the container with id 75cfbe0d95a14522f3ca27cdb0beb1709896272a35ebeeca09ed9381a9c7d0a2
Sep 07 13:14:03.021456 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:03.021365    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4" event={"ID":"9c609e16-b318-4b80-8113-15a6ebd303ba","Type":"ContainerStarted","Data":"043fa1f27e446f591c9a6b0f16f7cffa40c1f88d7c75798fa5cb22716b69a885"}
Sep 07 13:14:03.021456 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:03.021407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4" event={"ID":"9c609e16-b318-4b80-8113-15a6ebd303ba","Type":"ContainerStarted","Data":"75cfbe0d95a14522f3ca27cdb0beb1709896272a35ebeeca09ed9381a9c7d0a2"}
Sep 07 13:14:03.043369 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:03.043295    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4" podStartSLOduration=2.043279295 podStartE2EDuration="2.043279295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:03.040134758 +0000 UTC m=+1260.071001061" watchObservedRunningTime="2026-09-07 13:14:03.043279295 +0000 UTC m=+1260.074145605"
Sep 07 13:14:03.579817 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:03.579788    2565 scope.go:117] "RemoveContainer" containerID="e5e60bf37322cebb990d04000de24e6f8b9d44c2eb59c2c141cdf2ef2cf47b97"
Sep 07 13:14:03.589010 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:03.588987    2565 scope.go:117] "RemoveContainer" containerID="66096f166ebd47530ed92ec378db343a92487c4faed7580c2b69bdab735c238d"
Sep 07 13:14:03.909389 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:03.909292    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:14:12.323191 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:12.323152    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:12.323646 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:12.323252    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:12.335710 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:12.335688    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:13.090202 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:13.090170    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:14:23.705962 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:23.705915    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:14:43.754548 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:43.754511    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz"]
Sep 07 13:14:43.758425 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:43.758404    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz"
Sep 07 13:14:43.761797 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:43.761773    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flngs\""
Sep 07 13:14:43.761908 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:43.761894    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 13:14:43.770571 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:43.770499    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz"]
Sep 07 13:14:43.825653 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:43.825608    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad211ea-5903-4ce8-bfc6-9754dee612a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz\" (UID: \"2ad211ea-5903-4ce8-bfc6-9754dee612a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz"
Sep 07 13:14:43.927291 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:43.927252    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad211ea-5903-4ce8-bfc6-9754dee612a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz\" (UID: \"2ad211ea-5903-4ce8-bfc6-9754dee612a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz"
Sep 07 13:14:43.927997 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:43.927975    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad211ea-5903-4ce8-bfc6-9754dee612a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz\" (UID: \"2ad211ea-5903-4ce8-bfc6-9754dee612a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz"
Sep 07 13:14:44.071465 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:44.071429    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz"
Sep 07 13:14:44.351079 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:44.351051    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz"]
Sep 07 13:14:44.353883 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:14:44.353855    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad211ea_5903_4ce8_bfc6_9754dee612a9.slice/crio-78caace823f28bdcd651039a0ea9aa1dabda825e3d3b85e77da4568a03fc1850 WatchSource:0}: Error finding container 78caace823f28bdcd651039a0ea9aa1dabda825e3d3b85e77da4568a03fc1850: Status 404 returned error can't find the container with id 78caace823f28bdcd651039a0ea9aa1dabda825e3d3b85e77da4568a03fc1850
Sep 07 13:14:45.257670 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:45.257634    2565 generic.go:356] "Generic (PLEG): container finished" podID="2ad211ea-5903-4ce8-bfc6-9754dee612a9" containerID="7619620d123d7469ba1f4fb07ad74ecef983d769cb3380f513d8071a956e437c" exitCode=0
Sep 07 13:14:45.258111 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:45.257715    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz" event={"ID":"2ad211ea-5903-4ce8-bfc6-9754dee612a9","Type":"ContainerDied","Data":"7619620d123d7469ba1f4fb07ad74ecef983d769cb3380f513d8071a956e437c"}
Sep 07 13:14:45.258111 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:45.257767    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz" event={"ID":"2ad211ea-5903-4ce8-bfc6-9754dee612a9","Type":"ContainerStarted","Data":"78caace823f28bdcd651039a0ea9aa1dabda825e3d3b85e77da4568a03fc1850"}
Sep 07 13:14:46.308202 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:46.308166    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:14:46.398006 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:46.397980    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz"
Sep 07 13:14:46.454217 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:46.454183    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad211ea-5903-4ce8-bfc6-9754dee612a9-maas-api-service-ca\") pod \"2ad211ea-5903-4ce8-bfc6-9754dee612a9\" (UID: \"2ad211ea-5903-4ce8-bfc6-9754dee612a9\") "
Sep 07 13:14:46.454576 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:46.454549    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad211ea-5903-4ce8-bfc6-9754dee612a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ad211ea-5903-4ce8-bfc6-9754dee612a9" (UID: "2ad211ea-5903-4ce8-bfc6-9754dee612a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:14:46.555152 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:46.555116    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad211ea-5903-4ce8-bfc6-9754dee612a9-maas-api-service-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:14:47.269484 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:47.269452    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz"
Sep 07 13:14:47.269691 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:47.269496    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz" event={"ID":"2ad211ea-5903-4ce8-bfc6-9754dee612a9","Type":"ContainerDied","Data":"78caace823f28bdcd651039a0ea9aa1dabda825e3d3b85e77da4568a03fc1850"}
Sep 07 13:14:47.269691 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:47.269541    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78caace823f28bdcd651039a0ea9aa1dabda825e3d3b85e77da4568a03fc1850"
Sep 07 13:14:59.754012 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:59.753970    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz"]
Sep 07 13:14:59.757400 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:14:59.757372    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0561b5ad-cd6nz"]
Sep 07 13:15:00.152917 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:00.152875    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aa813e-29813115-x5dwj"]
Sep 07 13:15:00.153523 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:00.153503    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ad211ea-5903-4ce8-bfc6-9754dee612a9" containerName="revoke-keys"
Sep 07 13:15:00.153523 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:00.153524    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ad211ea-5903-4ce8-bfc6-9754dee612a9" containerName="revoke-keys"
Sep 07 13:15:00.153696 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:00.153647    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ad211ea-5903-4ce8-bfc6-9754dee612a9" containerName="revoke-keys"
Sep 07 13:15:00.157114 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:00.157092    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc4b05-29813115-dlbjd"]
Sep 07 13:15:00.157238 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:00.157214    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aa813e-29813115-x5dwj"
Sep 07 13:15:00.160982 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:00.160961    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc4b05-29813115-dlbjd"
Sep 07 13:15:00.162729 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:00.162709    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flngs\""
Sep 07 13:15:00.170388 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:00.170350    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aa813e-29813115-x5dwj"
Sep 07 13:15:00.174051 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:00.173828    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aa813e-29813115-x5dwj"]
Sep 07 13:15:00.177021 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:00.176992    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc4b05-29813115-dlbjd"]
Sep 07 13:15:00.181933 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:00.181910    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc4b05-29813115-dlbjd"
Sep 07 13:15:00.462514 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:00.462485    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aa813e-29813115-x5dwj"]
Sep 07 13:15:00.464565 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:15:00.464537    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961bd92b_ace0_488c_8b85_c3eaeb62f239.slice/crio-763ed243320aa6a2b717891f7034d0eff12670debb9f57d2232fcfc3e09d875b WatchSource:0}: Error finding container 763ed243320aa6a2b717891f7034d0eff12670debb9f57d2232fcfc3e09d875b: Status 404 returned error can't find the container with id 763ed243320aa6a2b717891f7034d0eff12670debb9f57d2232fcfc3e09d875b
Sep 07 13:15:00.473792 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:00.473765    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc4b05-29813115-dlbjd"]
Sep 07 13:15:00.478315 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:15:00.478284    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3043b73_1ff8_4426_90d4_d140dab602dc.slice/crio-df7b1e40571220b8d98be7d96bd4e535e28d177c89d1273a065dfa9e568850b8 WatchSource:0}: Error finding container df7b1e40571220b8d98be7d96bd4e535e28d177c89d1273a065dfa9e568850b8: Status 404 returned error can't find the container with id df7b1e40571220b8d98be7d96bd4e535e28d177c89d1273a065dfa9e568850b8
Sep 07 13:15:01.362188 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:01.362146    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc4b05-29813115-dlbjd" event={"ID":"b3043b73-1ff8-4426-90d4-d140dab602dc","Type":"ContainerStarted","Data":"a16e117379e4e7e62466f74e4362233ed3e6b1586a302b84ba3f5a1c44d65893"}
Sep 07 13:15:01.362764 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:01.362246    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc4b05-29813115-dlbjd" event={"ID":"b3043b73-1ff8-4426-90d4-d140dab602dc","Type":"ContainerStarted","Data":"df7b1e40571220b8d98be7d96bd4e535e28d177c89d1273a065dfa9e568850b8"}
Sep 07 13:15:01.363777 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:01.363747    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aa813e-29813115-x5dwj" event={"ID":"961bd92b-ace0-488c-8b85-c3eaeb62f239","Type":"ContainerStarted","Data":"4e9fb47ac6102a15a2d9bb22fe3acf004222c161e1169d8f2be0317d8f7b56f4"}
Sep 07 13:15:01.363886 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:01.363792    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aa813e-29813115-x5dwj" event={"ID":"961bd92b-ace0-488c-8b85-c3eaeb62f239","Type":"ContainerStarted","Data":"763ed243320aa6a2b717891f7034d0eff12670debb9f57d2232fcfc3e09d875b"}
Sep 07 13:15:01.379935 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:01.379887    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc4b05-29813115-dlbjd" podStartSLOduration=1.379873578 podStartE2EDuration="1.379873578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:15:01.378656174 +0000 UTC m=+1318.409522485" watchObservedRunningTime="2026-09-07 13:15:01.379873578 +0000 UTC m=+1318.410739887"
Sep 07 13:15:01.398019 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:01.397966    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aa813e-29813115-x5dwj" podStartSLOduration=1.397950142 podStartE2EDuration="1.397950142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:15:01.396526723 +0000 UTC m=+1318.427393032" watchObservedRunningTime="2026-09-07 13:15:01.397950142 +0000 UTC m=+1318.428816453"
Sep 07 13:15:01.489765 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:01.489732    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ad211ea-5903-4ce8-bfc6-9754dee612a9" path="/var/lib/kubelet/pods/2ad211ea-5903-4ce8-bfc6-9754dee612a9/volumes"
Sep 07 13:15:02.370335 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:02.370300    2565 generic.go:356] "Generic (PLEG): container finished" podID="b3043b73-1ff8-4426-90d4-d140dab602dc" containerID="a16e117379e4e7e62466f74e4362233ed3e6b1586a302b84ba3f5a1c44d65893" exitCode=0
Sep 07 13:15:02.370787 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:02.370376    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc4b05-29813115-dlbjd" event={"ID":"b3043b73-1ff8-4426-90d4-d140dab602dc","Type":"ContainerDied","Data":"a16e117379e4e7e62466f74e4362233ed3e6b1586a302b84ba3f5a1c44d65893"}
Sep 07 13:15:02.371841 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:02.371819    2565 generic.go:356] "Generic (PLEG): container finished" podID="961bd92b-ace0-488c-8b85-c3eaeb62f239" containerID="4e9fb47ac6102a15a2d9bb22fe3acf004222c161e1169d8f2be0317d8f7b56f4" exitCode=0
Sep 07 13:15:02.371938 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:02.371849    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aa813e-29813115-x5dwj" event={"ID":"961bd92b-ace0-488c-8b85-c3eaeb62f239","Type":"ContainerDied","Data":"4e9fb47ac6102a15a2d9bb22fe3acf004222c161e1169d8f2be0317d8f7b56f4"}
Sep 07 13:15:03.519763 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:03.519464    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aa813e-29813115-x5dwj"
Sep 07 13:15:03.556812 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:03.556786    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc4b05-29813115-dlbjd"
Sep 07 13:15:03.614791 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:03.614757    2565 scope.go:117] "RemoveContainer" containerID="4b810211d81dd013636e7a1683838ffa738e177765d61641adae973eebcf61c3"
Sep 07 13:15:04.384621 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:04.384588    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aa813e-29813115-x5dwj"
Sep 07 13:15:04.384814 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:04.384629    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aa813e-29813115-x5dwj" event={"ID":"961bd92b-ace0-488c-8b85-c3eaeb62f239","Type":"ContainerDied","Data":"763ed243320aa6a2b717891f7034d0eff12670debb9f57d2232fcfc3e09d875b"}
Sep 07 13:15:04.384814 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:04.384674    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="763ed243320aa6a2b717891f7034d0eff12670debb9f57d2232fcfc3e09d875b"
Sep 07 13:15:04.386027 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:04.386002    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc4b05-29813115-dlbjd" event={"ID":"b3043b73-1ff8-4426-90d4-d140dab602dc","Type":"ContainerDied","Data":"df7b1e40571220b8d98be7d96bd4e535e28d177c89d1273a065dfa9e568850b8"}
Sep 07 13:15:04.386136 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:04.386036    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc4b05-29813115-dlbjd"
Sep 07 13:15:04.386136 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:04.386049    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df7b1e40571220b8d98be7d96bd4e535e28d177c89d1273a065dfa9e568850b8"
Sep 07 13:15:06.039399 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:06.039358    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:15:08.238866 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:08.238827    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"]
Sep 07 13:15:21.430690 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:21.430654    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"]
Sep 07 13:15:23.259419 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.259377    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4" podUID="9c609e16-b318-4b80-8113-15a6ebd303ba" containerName="main" containerID="cri-o://043fa1f27e446f591c9a6b0f16f7cffa40c1f88d7c75798fa5cb22716b69a885" gracePeriod=45
Sep 07 13:15:23.500768 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.500581    2565 generic.go:356] "Generic (PLEG): container finished" podID="9c609e16-b318-4b80-8113-15a6ebd303ba" containerID="043fa1f27e446f591c9a6b0f16f7cffa40c1f88d7c75798fa5cb22716b69a885" exitCode=0
Sep 07 13:15:23.500768 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.500654    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4" event={"ID":"9c609e16-b318-4b80-8113-15a6ebd303ba","Type":"ContainerDied","Data":"043fa1f27e446f591c9a6b0f16f7cffa40c1f88d7c75798fa5cb22716b69a885"}
Sep 07 13:15:23.500768 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.500704    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4" event={"ID":"9c609e16-b318-4b80-8113-15a6ebd303ba","Type":"ContainerDied","Data":"75cfbe0d95a14522f3ca27cdb0beb1709896272a35ebeeca09ed9381a9c7d0a2"}
Sep 07 13:15:23.500768 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.500721    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75cfbe0d95a14522f3ca27cdb0beb1709896272a35ebeeca09ed9381a9c7d0a2"
Sep 07 13:15:23.517057 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.516992    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:15:23.644408 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.644365    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-home\") pod \"9c609e16-b318-4b80-8113-15a6ebd303ba\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") "
Sep 07 13:15:23.645455 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.644425    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-model-cache\") pod \"9c609e16-b318-4b80-8113-15a6ebd303ba\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") "
Sep 07 13:15:23.645455 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.644627    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-dshm\") pod \"9c609e16-b318-4b80-8113-15a6ebd303ba\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") "
Sep 07 13:15:23.645455 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.644753    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c609e16-b318-4b80-8113-15a6ebd303ba" (UID: "9c609e16-b318-4b80-8113-15a6ebd303ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:15:23.645455 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.644875    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-home" (OuterVolumeSpecName: "home") pod "9c609e16-b318-4b80-8113-15a6ebd303ba" (UID: "9c609e16-b318-4b80-8113-15a6ebd303ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:15:23.645455 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.644945    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-tmp-dir\") pod \"9c609e16-b318-4b80-8113-15a6ebd303ba\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") "
Sep 07 13:15:23.645455 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.645005    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c609e16-b318-4b80-8113-15a6ebd303ba-tls-certs\") pod \"9c609e16-b318-4b80-8113-15a6ebd303ba\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") "
Sep 07 13:15:23.645455 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.645030    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzlds\" (UniqueName: \"kubernetes.io/projected/9c609e16-b318-4b80-8113-15a6ebd303ba-kube-api-access-vzlds\") pod \"9c609e16-b318-4b80-8113-15a6ebd303ba\" (UID: \"9c609e16-b318-4b80-8113-15a6ebd303ba\") "
Sep 07 13:15:23.645917 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.645250    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c609e16-b318-4b80-8113-15a6ebd303ba" (UID: "9c609e16-b318-4b80-8113-15a6ebd303ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:15:23.646263 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.646230    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-tmp-dir\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:15:23.646263 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.646265    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-home\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:15:23.646519 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.646281    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-model-cache\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:15:23.647555 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.647531    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-dshm" (OuterVolumeSpecName: "dshm") pod "9c609e16-b318-4b80-8113-15a6ebd303ba" (UID: "9c609e16-b318-4b80-8113-15a6ebd303ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:15:23.647870 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.647849    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c609e16-b318-4b80-8113-15a6ebd303ba-kube-api-access-vzlds" (OuterVolumeSpecName: "kube-api-access-vzlds") pod "9c609e16-b318-4b80-8113-15a6ebd303ba" (UID: "9c609e16-b318-4b80-8113-15a6ebd303ba"). InnerVolumeSpecName "kube-api-access-vzlds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:15:23.648468 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.648451    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c609e16-b318-4b80-8113-15a6ebd303ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c609e16-b318-4b80-8113-15a6ebd303ba" (UID: "9c609e16-b318-4b80-8113-15a6ebd303ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:15:23.746970 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.746939    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c609e16-b318-4b80-8113-15a6ebd303ba-dshm\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:15:23.746970 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.746966    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c609e16-b318-4b80-8113-15a6ebd303ba-tls-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:15:23.746970 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:23.746976    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzlds\" (UniqueName: \"kubernetes.io/projected/9c609e16-b318-4b80-8113-15a6ebd303ba-kube-api-access-vzlds\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:15:24.505713 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:24.505680    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"
Sep 07 13:15:24.530248 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:24.530210    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"]
Sep 07 13:15:24.534973 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:24.534946    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-0561b5ad/e2e-lifecycle-model-0561b5ad-kserve-6c7fdb6cdc-hlsl4"]
Sep 07 13:15:24.601820 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:24.601786    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:15:25.489183 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:25.489105    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c609e16-b318-4b80-8113-15a6ebd303ba" path="/var/lib/kubelet/pods/9c609e16-b318-4b80-8113-15a6ebd303ba/volumes"
Sep 07 13:15:27.146785 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.146749    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"]
Sep 07 13:15:27.147297 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.147282    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3043b73-1ff8-4426-90d4-d140dab602dc" containerName="cleanup"
Sep 07 13:15:27.147363 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.147302    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3043b73-1ff8-4426-90d4-d140dab602dc" containerName="cleanup"
Sep 07 13:15:27.147363 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.147353    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c609e16-b318-4b80-8113-15a6ebd303ba" containerName="main"
Sep 07 13:15:27.147363 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.147362    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c609e16-b318-4b80-8113-15a6ebd303ba" containerName="main"
Sep 07 13:15:27.147472 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.147368    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="961bd92b-ace0-488c-8b85-c3eaeb62f239" containerName="cleanup"
Sep 07 13:15:27.147472 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.147373    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="961bd92b-ace0-488c-8b85-c3eaeb62f239" containerName="cleanup"
Sep 07 13:15:27.147472 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.147448    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3043b73-1ff8-4426-90d4-d140dab602dc" containerName="cleanup"
Sep 07 13:15:27.147472 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.147457    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c609e16-b318-4b80-8113-15a6ebd303ba" containerName="main"
Sep 07 13:15:27.147472 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.147463    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="961bd92b-ace0-488c-8b85-c3eaeb62f239" containerName="cleanup"
Sep 07 13:15:27.151806 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.151788    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.154987 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.154967    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-aa813e\"/\"sub-test-model-aa813e-kserve-self-signed-certs\""
Sep 07 13:15:27.169030 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.169004    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"]
Sep 07 13:15:27.287759 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.287722    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw27c\" (UniqueName: \"kubernetes.io/projected/0c46c909-16fb-4a00-88d5-66ec1b09e793-kube-api-access-rw27c\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.287759 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.287768    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-home\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.288018 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.287854    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-dshm\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.288018 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.287889    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-model-cache\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.288018 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.287940    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-tmp-dir\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.288018 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.287960    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c46c909-16fb-4a00-88d5-66ec1b09e793-tls-certs\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.389210 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.389174    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-dshm\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.389210 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.389212    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-model-cache\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.389518 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.389252    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-tmp-dir\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.389518 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.389272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c46c909-16fb-4a00-88d5-66ec1b09e793-tls-certs\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.389518 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.389362    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw27c\" (UniqueName: \"kubernetes.io/projected/0c46c909-16fb-4a00-88d5-66ec1b09e793-kube-api-access-rw27c\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.389518 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.389396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-home\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.389747 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.389610    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-model-cache\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.389747 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.389680    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-tmp-dir\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.389823 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.389777    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-home\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.391759 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.391731    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-dshm\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.391936 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.391919    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c46c909-16fb-4a00-88d5-66ec1b09e793-tls-certs\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.401607 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.401555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw27c\" (UniqueName: \"kubernetes.io/projected/0c46c909-16fb-4a00-88d5-66ec1b09e793-kube-api-access-rw27c\") pod \"sub-test-model-aa813e-kserve-844b7668b5-8rgmv\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.460618 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.460578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:27.735562 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:27.735537    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"]
Sep 07 13:15:27.737365 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:15:27.737318    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c46c909_16fb_4a00_88d5_66ec1b09e793.slice/crio-080118d8f06e539cd38268e360d5f673e7bada9a9b2dd4492cabec45aad50067 WatchSource:0}: Error finding container 080118d8f06e539cd38268e360d5f673e7bada9a9b2dd4492cabec45aad50067: Status 404 returned error can't find the container with id 080118d8f06e539cd38268e360d5f673e7bada9a9b2dd4492cabec45aad50067
Sep 07 13:15:28.534205 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:28.534171    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv" event={"ID":"0c46c909-16fb-4a00-88d5-66ec1b09e793","Type":"ContainerStarted","Data":"369d9cefe48174beca7580c1c5b6b2ae21a905c89d064e6d6ff0044a612bb7c0"}
Sep 07 13:15:28.534600 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:28.534214    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv" event={"ID":"0c46c909-16fb-4a00-88d5-66ec1b09e793","Type":"ContainerStarted","Data":"080118d8f06e539cd38268e360d5f673e7bada9a9b2dd4492cabec45aad50067"}
Sep 07 13:15:28.556639 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:28.556593    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv" podStartSLOduration=1.5565766239999999 podStartE2EDuration="1.556576624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:15:28.553472553 +0000 UTC m=+1345.584338863" watchObservedRunningTime="2026-09-07 13:15:28.556576624 +0000 UTC m=+1345.587442933"
Sep 07 13:15:36.451402 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.451341    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q" podUID="028abb1f-67e9-43bc-9c65-44456ab0fedf" containerName="main" containerID="cri-o://ec0cd26f1446beb577de9c8780913116a6a426c3bf7dde43666ff9dd6a664065" gracePeriod=45
Sep 07 13:15:36.589610 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.589576    2565 generic.go:356] "Generic (PLEG): container finished" podID="028abb1f-67e9-43bc-9c65-44456ab0fedf" containerID="ec0cd26f1446beb577de9c8780913116a6a426c3bf7dde43666ff9dd6a664065" exitCode=0
Sep 07 13:15:36.589772 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.589621    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q" event={"ID":"028abb1f-67e9-43bc-9c65-44456ab0fedf","Type":"ContainerDied","Data":"ec0cd26f1446beb577de9c8780913116a6a426c3bf7dde43666ff9dd6a664065"}
Sep 07 13:15:36.727741 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.727717    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:15:36.781156 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.781123    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fdgd\" (UniqueName: \"kubernetes.io/projected/028abb1f-67e9-43bc-9c65-44456ab0fedf-kube-api-access-7fdgd\") pod \"028abb1f-67e9-43bc-9c65-44456ab0fedf\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") "
Sep 07 13:15:36.781312 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.781189    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028abb1f-67e9-43bc-9c65-44456ab0fedf-tls-certs\") pod \"028abb1f-67e9-43bc-9c65-44456ab0fedf\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") "
Sep 07 13:15:36.781312 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.781217    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-dshm\") pod \"028abb1f-67e9-43bc-9c65-44456ab0fedf\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") "
Sep 07 13:15:36.781463 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.781360    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-home\") pod \"028abb1f-67e9-43bc-9c65-44456ab0fedf\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") "
Sep 07 13:15:36.781463 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.781425    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-tmp-dir\") pod \"028abb1f-67e9-43bc-9c65-44456ab0fedf\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") "
Sep 07 13:15:36.781578 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.781485    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-model-cache\") pod \"028abb1f-67e9-43bc-9c65-44456ab0fedf\" (UID: \"028abb1f-67e9-43bc-9c65-44456ab0fedf\") "
Sep 07 13:15:36.781670 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.781616    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-home" (OuterVolumeSpecName: "home") pod "028abb1f-67e9-43bc-9c65-44456ab0fedf" (UID: "028abb1f-67e9-43bc-9c65-44456ab0fedf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:15:36.781741 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.781684    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "028abb1f-67e9-43bc-9c65-44456ab0fedf" (UID: "028abb1f-67e9-43bc-9c65-44456ab0fedf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:15:36.781741 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.781723    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-model-cache" (OuterVolumeSpecName: "model-cache") pod "028abb1f-67e9-43bc-9c65-44456ab0fedf" (UID: "028abb1f-67e9-43bc-9c65-44456ab0fedf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:15:36.782135 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.782115    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-home\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:15:36.782183 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.782142    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-tmp-dir\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:15:36.782183 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.782158    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-model-cache\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:15:36.783680 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.783653    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028abb1f-67e9-43bc-9c65-44456ab0fedf-kube-api-access-7fdgd" (OuterVolumeSpecName: "kube-api-access-7fdgd") pod "028abb1f-67e9-43bc-9c65-44456ab0fedf" (UID: "028abb1f-67e9-43bc-9c65-44456ab0fedf"). InnerVolumeSpecName "kube-api-access-7fdgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:15:36.783938 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.783916    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-dshm" (OuterVolumeSpecName: "dshm") pod "028abb1f-67e9-43bc-9c65-44456ab0fedf" (UID: "028abb1f-67e9-43bc-9c65-44456ab0fedf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:15:36.783995 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.783917    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/028abb1f-67e9-43bc-9c65-44456ab0fedf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "028abb1f-67e9-43bc-9c65-44456ab0fedf" (UID: "028abb1f-67e9-43bc-9c65-44456ab0fedf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:15:36.883397 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.883359    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fdgd\" (UniqueName: \"kubernetes.io/projected/028abb1f-67e9-43bc-9c65-44456ab0fedf-kube-api-access-7fdgd\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:15:36.883397 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.883390    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028abb1f-67e9-43bc-9c65-44456ab0fedf-tls-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:15:36.883397 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:36.883401    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028abb1f-67e9-43bc-9c65-44456ab0fedf-dshm\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:15:37.461193 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:37.461155    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:37.461717 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:37.461216    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:37.473796 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:37.473769    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:37.596507 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:37.596467    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q" event={"ID":"028abb1f-67e9-43bc-9c65-44456ab0fedf","Type":"ContainerDied","Data":"abfb2feec3545b256142b97bf0fcc518039d778ec8e2f0e2fc5afe2a0deaf49d"}
Sep 07 13:15:37.596698 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:37.596537    2565 scope.go:117] "RemoveContainer" containerID="ec0cd26f1446beb577de9c8780913116a6a426c3bf7dde43666ff9dd6a664065"
Sep 07 13:15:37.596698 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:37.596491    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"
Sep 07 13:15:37.608456 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:37.608433    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:15:37.617926 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:37.617900    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"]
Sep 07 13:15:37.623003 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:37.622981    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/auth-test-model-aa813e-kserve-579cc9b687-5h97q"]
Sep 07 13:15:39.489393 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:39.489358    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="028abb1f-67e9-43bc-9c65-44456ab0fedf" path="/var/lib/kubelet/pods/028abb1f-67e9-43bc-9c65-44456ab0fedf/volumes"
Sep 07 13:15:40.355824 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:40.355786    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:15:58.117848 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:15:58.117805    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:16:01.263057 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:01.263026    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"]
Sep 07 13:16:01.263491 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:01.263266    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g" podUID="ad31d994-184e-4491-8f55-8f0208e5a127" containerName="istio-proxy" containerID="cri-o://79cec92a46b8c1c05cdfee7a280b5a5060adb1cfa336f1c76e1d553c711ac54a" gracePeriod=30
Sep 07 13:16:03.637166 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:03.637127    2565 scope.go:117] "RemoveContainer" containerID="379c89ae336700a86a294ff0ccc275d45e55f35a451649e1144feec86897b718"
Sep 07 13:16:06.526801 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.526736    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:16:06.588315 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.588270    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-istio-envoy\") pod \"ad31d994-184e-4491-8f55-8f0208e5a127\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") "
Sep 07 13:16:06.588315 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.588352    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad31d994-184e-4491-8f55-8f0208e5a127-istio-token\") pod \"ad31d994-184e-4491-8f55-8f0208e5a127\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") "
Sep 07 13:16:06.588616 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.588379    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvqpr\" (UniqueName: \"kubernetes.io/projected/ad31d994-184e-4491-8f55-8f0208e5a127-kube-api-access-zvqpr\") pod \"ad31d994-184e-4491-8f55-8f0208e5a127\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") "
Sep 07 13:16:06.588616 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.588395    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-workload-certs\") pod \"ad31d994-184e-4491-8f55-8f0208e5a127\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") "
Sep 07 13:16:06.588616 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.588424    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad31d994-184e-4491-8f55-8f0208e5a127-istio-podinfo\") pod \"ad31d994-184e-4491-8f55-8f0208e5a127\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") "
Sep 07 13:16:06.588616 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.588446    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad31d994-184e-4491-8f55-8f0208e5a127-istiod-ca-cert\") pod \"ad31d994-184e-4491-8f55-8f0208e5a127\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") "
Sep 07 13:16:06.588616 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.588492    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-istio-data\") pod \"ad31d994-184e-4491-8f55-8f0208e5a127\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") "
Sep 07 13:16:06.588616 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.588519    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-workload-socket\") pod \"ad31d994-184e-4491-8f55-8f0208e5a127\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") "
Sep 07 13:16:06.588616 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.588556    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-credential-socket\") pod \"ad31d994-184e-4491-8f55-8f0208e5a127\" (UID: \"ad31d994-184e-4491-8f55-8f0208e5a127\") "
Sep 07 13:16:06.588992 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.588731    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ad31d994-184e-4491-8f55-8f0208e5a127" (UID: "ad31d994-184e-4491-8f55-8f0208e5a127"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:16:06.588992 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.588942    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ad31d994-184e-4491-8f55-8f0208e5a127" (UID: "ad31d994-184e-4491-8f55-8f0208e5a127"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:16:06.589161 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.588997    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-workload-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:16:06.589161 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.589156    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ad31d994-184e-4491-8f55-8f0208e5a127" (UID: "ad31d994-184e-4491-8f55-8f0208e5a127"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:16:06.589403 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.589188    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-istio-data" (OuterVolumeSpecName: "istio-data") pod "ad31d994-184e-4491-8f55-8f0208e5a127" (UID: "ad31d994-184e-4491-8f55-8f0208e5a127"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:16:06.589403 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.589383    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad31d994-184e-4491-8f55-8f0208e5a127-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ad31d994-184e-4491-8f55-8f0208e5a127" (UID: "ad31d994-184e-4491-8f55-8f0208e5a127"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:16:06.590739 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.590715    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ad31d994-184e-4491-8f55-8f0208e5a127" (UID: "ad31d994-184e-4491-8f55-8f0208e5a127"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:16:06.590921 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.590900    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad31d994-184e-4491-8f55-8f0208e5a127-kube-api-access-zvqpr" (OuterVolumeSpecName: "kube-api-access-zvqpr") pod "ad31d994-184e-4491-8f55-8f0208e5a127" (UID: "ad31d994-184e-4491-8f55-8f0208e5a127"). InnerVolumeSpecName "kube-api-access-zvqpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:16:06.591019 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.590954    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad31d994-184e-4491-8f55-8f0208e5a127-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ad31d994-184e-4491-8f55-8f0208e5a127" (UID: "ad31d994-184e-4491-8f55-8f0208e5a127"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 13:16:06.591019 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.590969    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad31d994-184e-4491-8f55-8f0208e5a127-istio-token" (OuterVolumeSpecName: "istio-token") pod "ad31d994-184e-4491-8f55-8f0208e5a127" (UID: "ad31d994-184e-4491-8f55-8f0208e5a127"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:16:06.690017 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.689985    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-istio-envoy\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:16:06.690017 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.690012    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad31d994-184e-4491-8f55-8f0208e5a127-istio-token\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:16:06.690017 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.690021    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvqpr\" (UniqueName: \"kubernetes.io/projected/ad31d994-184e-4491-8f55-8f0208e5a127-kube-api-access-zvqpr\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:16:06.690255 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.690031    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad31d994-184e-4491-8f55-8f0208e5a127-istio-podinfo\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:16:06.690255 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.690040    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad31d994-184e-4491-8f55-8f0208e5a127-istiod-ca-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:16:06.690255 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.690049    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-istio-data\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:16:06.690255 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.690056    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-workload-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:16:06.690255 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.690064    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad31d994-184e-4491-8f55-8f0208e5a127-credential-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:16:06.762787 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.762754    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad31d994-184e-4491-8f55-8f0208e5a127" containerID="79cec92a46b8c1c05cdfee7a280b5a5060adb1cfa336f1c76e1d553c711ac54a" exitCode=0
Sep 07 13:16:06.762993 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.762808    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"
Sep 07 13:16:06.762993 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.762851    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g" event={"ID":"ad31d994-184e-4491-8f55-8f0208e5a127","Type":"ContainerDied","Data":"79cec92a46b8c1c05cdfee7a280b5a5060adb1cfa336f1c76e1d553c711ac54a"}
Sep 07 13:16:06.762993 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.762917    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g" event={"ID":"ad31d994-184e-4491-8f55-8f0208e5a127","Type":"ContainerDied","Data":"f3a7b2e3df3b243b6817f9f5dcb63cec30aea17dc6c54097e05170adc93fc68b"}
Sep 07 13:16:06.762993 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.762945    2565 scope.go:117] "RemoveContainer" containerID="79cec92a46b8c1c05cdfee7a280b5a5060adb1cfa336f1c76e1d553c711ac54a"
Sep 07 13:16:06.772704 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.772687    2565 scope.go:117] "RemoveContainer" containerID="cfa6b9e6ed0c3a18b8461dcee3c2683c6153d5097ba6c4afd3e09ae59d7c954e"
Sep 07 13:16:06.781351 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.781316    2565 scope.go:117] "RemoveContainer" containerID="79cec92a46b8c1c05cdfee7a280b5a5060adb1cfa336f1c76e1d553c711ac54a"
Sep 07 13:16:06.781611 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:16:06.781593    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79cec92a46b8c1c05cdfee7a280b5a5060adb1cfa336f1c76e1d553c711ac54a\": container with ID starting with 79cec92a46b8c1c05cdfee7a280b5a5060adb1cfa336f1c76e1d553c711ac54a not found: ID does not exist" containerID="79cec92a46b8c1c05cdfee7a280b5a5060adb1cfa336f1c76e1d553c711ac54a"
Sep 07 13:16:06.781664 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.781620    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79cec92a46b8c1c05cdfee7a280b5a5060adb1cfa336f1c76e1d553c711ac54a"} err="failed to get container status \"79cec92a46b8c1c05cdfee7a280b5a5060adb1cfa336f1c76e1d553c711ac54a\": rpc error: code = NotFound desc = could not find container \"79cec92a46b8c1c05cdfee7a280b5a5060adb1cfa336f1c76e1d553c711ac54a\": container with ID starting with 79cec92a46b8c1c05cdfee7a280b5a5060adb1cfa336f1c76e1d553c711ac54a not found: ID does not exist"
Sep 07 13:16:06.781664 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.781637    2565 scope.go:117] "RemoveContainer" containerID="cfa6b9e6ed0c3a18b8461dcee3c2683c6153d5097ba6c4afd3e09ae59d7c954e"
Sep 07 13:16:06.781879 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:16:06.781861    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa6b9e6ed0c3a18b8461dcee3c2683c6153d5097ba6c4afd3e09ae59d7c954e\": container with ID starting with cfa6b9e6ed0c3a18b8461dcee3c2683c6153d5097ba6c4afd3e09ae59d7c954e not found: ID does not exist" containerID="cfa6b9e6ed0c3a18b8461dcee3c2683c6153d5097ba6c4afd3e09ae59d7c954e"
Sep 07 13:16:06.781921 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.781885    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa6b9e6ed0c3a18b8461dcee3c2683c6153d5097ba6c4afd3e09ae59d7c954e"} err="failed to get container status \"cfa6b9e6ed0c3a18b8461dcee3c2683c6153d5097ba6c4afd3e09ae59d7c954e\": rpc error: code = NotFound desc = could not find container \"cfa6b9e6ed0c3a18b8461dcee3c2683c6153d5097ba6c4afd3e09ae59d7c954e\": container with ID starting with cfa6b9e6ed0c3a18b8461dcee3c2683c6153d5097ba6c4afd3e09ae59d7c954e not found: ID does not exist"
Sep 07 13:16:06.787818 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.787798    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"]
Sep 07 13:16:06.792807 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:06.792785    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0561b5ad-openshift-default-665649f664-w668g"]
Sep 07 13:16:07.488184 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:07.488150    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad31d994-184e-4491-8f55-8f0208e5a127" path="/var/lib/kubelet/pods/ad31d994-184e-4491-8f55-8f0208e5a127/volumes"
Sep 07 13:16:16.836830 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:16.836792    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:16:36.518263 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:36.518228    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:16:38.703691 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:38.703648    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"]
Sep 07 13:16:50.026070 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.026028    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"]
Sep 07 13:16:50.026696 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.026512    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad31d994-184e-4491-8f55-8f0208e5a127" containerName="istio-proxy"
Sep 07 13:16:50.026696 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.026526    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad31d994-184e-4491-8f55-8f0208e5a127" containerName="istio-proxy"
Sep 07 13:16:50.026696 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.026552    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="028abb1f-67e9-43bc-9c65-44456ab0fedf" containerName="main"
Sep 07 13:16:50.026696 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.026558    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="028abb1f-67e9-43bc-9c65-44456ab0fedf" containerName="main"
Sep 07 13:16:50.026696 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.026622    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad31d994-184e-4491-8f55-8f0208e5a127" containerName="istio-proxy"
Sep 07 13:16:50.026696 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.026631    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad31d994-184e-4491-8f55-8f0208e5a127" containerName="istio-proxy"
Sep 07 13:16:50.026696 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.026639    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="028abb1f-67e9-43bc-9c65-44456ab0fedf" containerName="main"
Sep 07 13:16:50.029907 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.029887    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.032677 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.032657    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-aa813e\"/\"rate-test-model-aa813e-kserve-self-signed-certs\""
Sep 07 13:16:50.040723 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.040698    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"]
Sep 07 13:16:50.130757 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.130718    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvbf4\" (UniqueName: \"kubernetes.io/projected/641b1958-7356-4bd2-add2-a7a7afaa8713-kube-api-access-lvbf4\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.130937 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.130793    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-model-cache\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.130937 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.130837    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-tmp-dir\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.130937 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.130868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-home\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.130937 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.130896    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641b1958-7356-4bd2-add2-a7a7afaa8713-tls-certs\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.130937 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.130923    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-dshm\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.232274 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.232238    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641b1958-7356-4bd2-add2-a7a7afaa8713-tls-certs\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.232499 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.232286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-dshm\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.232499 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.232341    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvbf4\" (UniqueName: \"kubernetes.io/projected/641b1958-7356-4bd2-add2-a7a7afaa8713-kube-api-access-lvbf4\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.232499 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.232381    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-model-cache\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.232499 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.232415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-tmp-dir\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.232499 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.232440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-home\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.232836 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.232813    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-model-cache\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.232880 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.232837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-home\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.232880 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.232871    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-tmp-dir\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.234819 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.234790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-dshm\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.234975 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.234935    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641b1958-7356-4bd2-add2-a7a7afaa8713-tls-certs\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.241898 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.241873    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvbf4\" (UniqueName: \"kubernetes.io/projected/641b1958-7356-4bd2-add2-a7a7afaa8713-kube-api-access-lvbf4\") pod \"rate-test-model-aa813e-kserve-7cc54cc847-5zds8\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") " pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.339796 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.339752    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:16:50.618633 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:50.618606    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"]
Sep 07 13:16:50.620920 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:16:50.620886    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod641b1958_7356_4bd2_add2_a7a7afaa8713.slice/crio-23f903e5f53d74d809dabb324139c5b5807090e1292fbd2bd810f00aaf91bd07 WatchSource:0}: Error finding container 23f903e5f53d74d809dabb324139c5b5807090e1292fbd2bd810f00aaf91bd07: Status 404 returned error can't find the container with id 23f903e5f53d74d809dabb324139c5b5807090e1292fbd2bd810f00aaf91bd07
Sep 07 13:16:51.017821 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:51.017726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8" event={"ID":"641b1958-7356-4bd2-add2-a7a7afaa8713","Type":"ContainerStarted","Data":"68b895313119f34072a5fe37a28d591dc2c2892fc4fd95aba1e82981d3ef5d7a"}
Sep 07 13:16:51.017821 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:51.017770    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8" event={"ID":"641b1958-7356-4bd2-add2-a7a7afaa8713","Type":"ContainerStarted","Data":"23f903e5f53d74d809dabb324139c5b5807090e1292fbd2bd810f00aaf91bd07"}
Sep 07 13:16:51.043273 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:51.043224    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8" podStartSLOduration=2.043203604 podStartE2EDuration="2.043203604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:16:51.041595217 +0000 UTC m=+1428.072461540" watchObservedRunningTime="2026-09-07 13:16:51.043203604 +0000 UTC m=+1428.074069914"
Sep 07 13:16:53.713046 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:53.713009    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:16:53.723859 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:53.723819    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv" podUID="0c46c909-16fb-4a00-88d5-66ec1b09e793" containerName="main" containerID="cri-o://369d9cefe48174beca7580c1c5b6b2ae21a905c89d064e6d6ff0044a612bb7c0" gracePeriod=45
Sep 07 13:16:53.980421 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:53.980396    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:16:54.035772 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.035735    2565 generic.go:356] "Generic (PLEG): container finished" podID="0c46c909-16fb-4a00-88d5-66ec1b09e793" containerID="369d9cefe48174beca7580c1c5b6b2ae21a905c89d064e6d6ff0044a612bb7c0" exitCode=0
Sep 07 13:16:54.035907 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.035781    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"
Sep 07 13:16:54.035907 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.035784    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv" event={"ID":"0c46c909-16fb-4a00-88d5-66ec1b09e793","Type":"ContainerDied","Data":"369d9cefe48174beca7580c1c5b6b2ae21a905c89d064e6d6ff0044a612bb7c0"}
Sep 07 13:16:54.035995 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.035905    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv" event={"ID":"0c46c909-16fb-4a00-88d5-66ec1b09e793","Type":"ContainerDied","Data":"080118d8f06e539cd38268e360d5f673e7bada9a9b2dd4492cabec45aad50067"}
Sep 07 13:16:54.035995 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.035929    2565 scope.go:117] "RemoveContainer" containerID="369d9cefe48174beca7580c1c5b6b2ae21a905c89d064e6d6ff0044a612bb7c0"
Sep 07 13:16:54.044742 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.044722    2565 scope.go:117] "RemoveContainer" containerID="369d9cefe48174beca7580c1c5b6b2ae21a905c89d064e6d6ff0044a612bb7c0"
Sep 07 13:16:54.045034 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:16:54.044994    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"369d9cefe48174beca7580c1c5b6b2ae21a905c89d064e6d6ff0044a612bb7c0\": container with ID starting with 369d9cefe48174beca7580c1c5b6b2ae21a905c89d064e6d6ff0044a612bb7c0 not found: ID does not exist" containerID="369d9cefe48174beca7580c1c5b6b2ae21a905c89d064e6d6ff0044a612bb7c0"
Sep 07 13:16:54.045077 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.045044    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"369d9cefe48174beca7580c1c5b6b2ae21a905c89d064e6d6ff0044a612bb7c0"} err="failed to get container status \"369d9cefe48174beca7580c1c5b6b2ae21a905c89d064e6d6ff0044a612bb7c0\": rpc error: code = NotFound desc = could not find container \"369d9cefe48174beca7580c1c5b6b2ae21a905c89d064e6d6ff0044a612bb7c0\": container with ID starting with 369d9cefe48174beca7580c1c5b6b2ae21a905c89d064e6d6ff0044a612bb7c0 not found: ID does not exist"
Sep 07 13:16:54.069373 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.069353    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-dshm\") pod \"0c46c909-16fb-4a00-88d5-66ec1b09e793\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") "
Sep 07 13:16:54.069476 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.069435    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw27c\" (UniqueName: \"kubernetes.io/projected/0c46c909-16fb-4a00-88d5-66ec1b09e793-kube-api-access-rw27c\") pod \"0c46c909-16fb-4a00-88d5-66ec1b09e793\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") "
Sep 07 13:16:54.069521 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.069477    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c46c909-16fb-4a00-88d5-66ec1b09e793-tls-certs\") pod \"0c46c909-16fb-4a00-88d5-66ec1b09e793\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") "
Sep 07 13:16:54.069561 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.069542    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-model-cache\") pod \"0c46c909-16fb-4a00-88d5-66ec1b09e793\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") "
Sep 07 13:16:54.069607 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.069577    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-tmp-dir\") pod \"0c46c909-16fb-4a00-88d5-66ec1b09e793\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") "
Sep 07 13:16:54.069668 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.069623    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-home\") pod \"0c46c909-16fb-4a00-88d5-66ec1b09e793\" (UID: \"0c46c909-16fb-4a00-88d5-66ec1b09e793\") "
Sep 07 13:16:54.069843 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.069818    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c46c909-16fb-4a00-88d5-66ec1b09e793" (UID: "0c46c909-16fb-4a00-88d5-66ec1b09e793"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:16:54.069931 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.069841    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c46c909-16fb-4a00-88d5-66ec1b09e793" (UID: "0c46c909-16fb-4a00-88d5-66ec1b09e793"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:16:54.070041 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.070017    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-home" (OuterVolumeSpecName: "home") pod "0c46c909-16fb-4a00-88d5-66ec1b09e793" (UID: "0c46c909-16fb-4a00-88d5-66ec1b09e793"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:16:54.070041 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.070034    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-model-cache\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:16:54.070231 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.070055    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-tmp-dir\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:16:54.071622 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.071597    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-dshm" (OuterVolumeSpecName: "dshm") pod "0c46c909-16fb-4a00-88d5-66ec1b09e793" (UID: "0c46c909-16fb-4a00-88d5-66ec1b09e793"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:16:54.071812 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.071792    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c46c909-16fb-4a00-88d5-66ec1b09e793-kube-api-access-rw27c" (OuterVolumeSpecName: "kube-api-access-rw27c") pod "0c46c909-16fb-4a00-88d5-66ec1b09e793" (UID: "0c46c909-16fb-4a00-88d5-66ec1b09e793"). InnerVolumeSpecName "kube-api-access-rw27c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:16:54.072076 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.072056    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c46c909-16fb-4a00-88d5-66ec1b09e793-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c46c909-16fb-4a00-88d5-66ec1b09e793" (UID: "0c46c909-16fb-4a00-88d5-66ec1b09e793"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:16:54.171540 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.171503    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c46c909-16fb-4a00-88d5-66ec1b09e793-tls-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:16:54.171540 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.171539    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-home\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:16:54.171540 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.171551    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c46c909-16fb-4a00-88d5-66ec1b09e793-dshm\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:16:54.171782 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.171559    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw27c\" (UniqueName: \"kubernetes.io/projected/0c46c909-16fb-4a00-88d5-66ec1b09e793-kube-api-access-rw27c\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:16:54.359581 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.359547    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"]
Sep 07 13:16:54.364222 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:54.364199    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/sub-test-model-aa813e-kserve-844b7668b5-8rgmv"]
Sep 07 13:16:55.490970 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:55.490893    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c46c909-16fb-4a00-88d5-66ec1b09e793" path="/var/lib/kubelet/pods/0c46c909-16fb-4a00-88d5-66ec1b09e793/volumes"
Sep 07 13:16:58.710074 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.710042    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"]
Sep 07 13:16:58.710523 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.710508    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad31d994-184e-4491-8f55-8f0208e5a127" containerName="istio-proxy"
Sep 07 13:16:58.710523 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.710524    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad31d994-184e-4491-8f55-8f0208e5a127" containerName="istio-proxy"
Sep 07 13:16:58.710614 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.710554    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c46c909-16fb-4a00-88d5-66ec1b09e793" containerName="main"
Sep 07 13:16:58.710614 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.710559    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c46c909-16fb-4a00-88d5-66ec1b09e793" containerName="main"
Sep 07 13:16:58.710698 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.710625    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c46c909-16fb-4a00-88d5-66ec1b09e793" containerName="main"
Sep 07 13:16:58.715110 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.715083    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.717694 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.717670    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ddb9ca-openshift-default-dockercfg-zdznx\""
Sep 07 13:16:58.728056 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.728031    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"]
Sep 07 13:16:58.816372 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.816306    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-token\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.816588 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.816383    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-credential-socket\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.816588 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.816440    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxv6z\" (UniqueName: \"kubernetes.io/projected/991813eb-2fb3-48aa-9908-b5af528eb5f5-kube-api-access-wxv6z\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.816588 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.816473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-data\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.816588 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.816509    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991813eb-2fb3-48aa-9908-b5af528eb5f5-istiod-ca-cert\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.816818 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.816618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-workload-socket\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.816818 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.816644    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-envoy\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.816818 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.816692    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-podinfo\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.816818 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.816735    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-workload-certs\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.917380 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.917292    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-credential-socket\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.917380 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.917384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxv6z\" (UniqueName: \"kubernetes.io/projected/991813eb-2fb3-48aa-9908-b5af528eb5f5-kube-api-access-wxv6z\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.917709 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.917411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-data\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.917709 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.917440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991813eb-2fb3-48aa-9908-b5af528eb5f5-istiod-ca-cert\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.917709 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.917514    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-workload-socket\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.917709 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.917541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-envoy\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.917709 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.917585    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-podinfo\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.917709 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.917627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-workload-certs\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.917709 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.917700    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-token\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.918085 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.917813    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-credential-socket\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.918085 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.917839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-data\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.918085 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.917883    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-workload-socket\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.918469 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.918199    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-workload-certs\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.918469 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.918292    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991813eb-2fb3-48aa-9908-b5af528eb5f5-istiod-ca-cert\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.920145 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.920122    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-envoy\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.920248 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.920229    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-podinfo\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.926726 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.926705    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxv6z\" (UniqueName: \"kubernetes.io/projected/991813eb-2fb3-48aa-9908-b5af528eb5f5-kube-api-access-wxv6z\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:58.926938 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:58.926918    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-token\") pod \"e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") " pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:59.025176 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:59.025081    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:16:59.304956 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:59.304865    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"]
Sep 07 13:16:59.307863 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:16:59.307829    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991813eb_2fb3_48aa_9908_b5af528eb5f5.slice/crio-b64857104c542f654892f7c8788a742f461123b3f69849e4d24437c63c77d06d WatchSource:0}: Error finding container b64857104c542f654892f7c8788a742f461123b3f69849e4d24437c63c77d06d: Status 404 returned error can't find the container with id b64857104c542f654892f7c8788a742f461123b3f69849e4d24437c63c77d06d
Sep 07 13:16:59.310105 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:59.310073    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:16:59.310185 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:59.310138    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:16:59.310224 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:16:59.310184    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:17:00.072542 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:00.072504    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z" event={"ID":"991813eb-2fb3-48aa-9908-b5af528eb5f5","Type":"ContainerStarted","Data":"e6a50a37ad0d414d8758bc08867c59d04a5295c50770b2d0d84c66f76fd88eaa"}
Sep 07 13:17:00.072919 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:00.072559    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z" event={"ID":"991813eb-2fb3-48aa-9908-b5af528eb5f5","Type":"ContainerStarted","Data":"b64857104c542f654892f7c8788a742f461123b3f69849e4d24437c63c77d06d"}
Sep 07 13:17:00.098748 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:00.098698    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z" podStartSLOduration=2.098681892 podStartE2EDuration="2.098681892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:17:00.096345913 +0000 UTC m=+1437.127212222" watchObservedRunningTime="2026-09-07 13:17:00.098681892 +0000 UTC m=+1437.129548202"
Sep 07 13:17:00.340890 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:00.340793    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:17:00.340890 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:00.340849    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:17:00.353657 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:00.353630    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:17:01.025606 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:01.025566    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:17:01.030737 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:01.030711    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:17:01.078362 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:01.078298    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:17:01.079100 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:01.079081    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:17:01.089360 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:01.089342    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:17:13.003676 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:13.003641    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:17:23.213831 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:23.213793    2565 generic.go:356] "Generic (PLEG): container finished" podID="991813eb-2fb3-48aa-9908-b5af528eb5f5" containerID="e6a50a37ad0d414d8758bc08867c59d04a5295c50770b2d0d84c66f76fd88eaa" exitCode=0
Sep 07 13:17:23.214242 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:23.213837    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z" event={"ID":"991813eb-2fb3-48aa-9908-b5af528eb5f5","Type":"ContainerDied","Data":"e6a50a37ad0d414d8758bc08867c59d04a5295c50770b2d0d84c66f76fd88eaa"}
Sep 07 13:17:23.214317 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:23.214295    2565 scope.go:117] "RemoveContainer" containerID="e6a50a37ad0d414d8758bc08867c59d04a5295c50770b2d0d84c66f76fd88eaa"
Sep 07 13:17:23.215507 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:23.215476    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:17:23.215568 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:23.215548    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:17:23.215604 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:23.215580    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:17:24.222056 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:24.222019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z" event={"ID":"991813eb-2fb3-48aa-9908-b5af528eb5f5","Type":"ContainerStarted","Data":"e1ac81725e61d673fc9e8b843e33b7d16f4aaa2927872a8e200d0158ec0f3b95"}
Sep 07 13:17:25.025723 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:25.025675    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:17:25.025950 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:25.025859    2565 patch_prober.go:28] interesting pod/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 07 13:17:25.025950 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:25.025919    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z" podUID="991813eb-2fb3-48aa-9908-b5af528eb5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 07 13:17:26.025725 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:26.025685    2565 patch_prober.go:28] interesting pod/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 07 13:17:26.026194 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:26.025748    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z" podUID="991813eb-2fb3-48aa-9908-b5af528eb5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 07 13:17:27.025978 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:27.025939    2565 patch_prober.go:28] interesting pod/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 07 13:17:27.026405 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:27.026000    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z" podUID="991813eb-2fb3-48aa-9908-b5af528eb5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 07 13:17:28.025801 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.025765    2565 patch_prober.go:28] interesting pod/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 07 13:17:28.025983 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.025822    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z" podUID="991813eb-2fb3-48aa-9908-b5af528eb5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 07 13:17:28.102353 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.100753    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"]
Sep 07 13:17:28.113150 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.113121    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.116399 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.116371    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0d2b4b-openshift-default-dockercfg-8xlvr\""
Sep 07 13:17:28.119732 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.119710    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"]
Sep 07 13:17:28.231432 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.231391    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-credential-socket\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.231615 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.231446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-data\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.231615 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.231492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-envoy\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.231615 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.231528    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-podinfo\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.231757 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.231607    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3bc808-8d70-4bbb-9e1f-88a278834552-istiod-ca-cert\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.231757 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.231659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-workload-socket\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.231757 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.231686    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-token\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.231757 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.231703    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-workload-certs\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.231757 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.231726    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwckm\" (UniqueName: \"kubernetes.io/projected/3c3bc808-8d70-4bbb-9e1f-88a278834552-kube-api-access-cwckm\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.332989 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.332954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-credential-socket\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.333192 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.333008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-data\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.333192 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.333035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-envoy\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.333192 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.333066    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-podinfo\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.333192 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.333097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3bc808-8d70-4bbb-9e1f-88a278834552-istiod-ca-cert\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.333192 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.333128    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-workload-socket\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.333192 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.333158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-token\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.333192 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.333182    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-workload-certs\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.333606 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.333407    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-credential-socket\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.333606 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.333457    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwckm\" (UniqueName: \"kubernetes.io/projected/3c3bc808-8d70-4bbb-9e1f-88a278834552-kube-api-access-cwckm\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.333840 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.333821    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-data\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.333928 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.333828    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3bc808-8d70-4bbb-9e1f-88a278834552-istiod-ca-cert\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.333928 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.333917    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-workload-certs\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.334004 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.333978    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-workload-socket\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.335663 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.335642    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-envoy\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.335759 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.335641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-podinfo\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.342604 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.342569    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-token\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.343152 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.343129    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwckm\" (UniqueName: \"kubernetes.io/projected/3c3bc808-8d70-4bbb-9e1f-88a278834552-kube-api-access-cwckm\") pod \"e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") " pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.427992 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.427949    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:28.733681 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.733581    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"]
Sep 07 13:17:28.739183 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:17:28.739158    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3bc808_8d70_4bbb_9e1f_88a278834552.slice/crio-64682800bca7986143d10f1f9892ee64695ef2a9b795621f358f72f47c155b86 WatchSource:0}: Error finding container 64682800bca7986143d10f1f9892ee64695ef2a9b795621f358f72f47c155b86: Status 404 returned error can't find the container with id 64682800bca7986143d10f1f9892ee64695ef2a9b795621f358f72f47c155b86
Sep 07 13:17:28.741673 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.741631    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:17:28.741754 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.741735    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:17:28.741802 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:28.741786    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:17:29.025961 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:29.025863    2565 patch_prober.go:28] interesting pod/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 07 13:17:29.025961 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:29.025918    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z" podUID="991813eb-2fb3-48aa-9908-b5af528eb5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 07 13:17:29.254275 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:29.254231    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g" event={"ID":"3c3bc808-8d70-4bbb-9e1f-88a278834552","Type":"ContainerStarted","Data":"bc823883e4d3d8690481ed7d4c5654b2db766a7a02253f254b5e0feb5b8bd31c"}
Sep 07 13:17:29.254683 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:29.254385    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g" event={"ID":"3c3bc808-8d70-4bbb-9e1f-88a278834552","Type":"ContainerStarted","Data":"64682800bca7986143d10f1f9892ee64695ef2a9b795621f358f72f47c155b86"}
Sep 07 13:17:29.281489 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:29.281380    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g" podStartSLOduration=1.2813636800000001 podStartE2EDuration="1.28136368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:17:29.279317538 +0000 UTC m=+1466.310183859" watchObservedRunningTime="2026-09-07 13:17:29.28136368 +0000 UTC m=+1466.312230021"
Sep 07 13:17:29.428575 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:29.428530    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:29.433793 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:29.433768    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:30.029637 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:30.029608    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:17:30.029891 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:30.029868    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:17:30.030521 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:30.030505    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:17:30.260530 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:30.260502    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:30.261500 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:30.261479    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:33.405882 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:33.405850    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"]
Sep 07 13:17:33.409538 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:33.409520    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"
Sep 07 13:17:33.412594 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:33.412568    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0d2b4b\""
Sep 07 13:17:33.412594 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:33.412568    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0d2b4b-dockercfg-cfsrv\""
Sep 07 13:17:33.421419 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:33.421385    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"]
Sep 07 13:17:33.494098 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:33.494039    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tfp9\" (UniqueName: \"kubernetes.io/projected/003fe297-627b-493b-a55b-cce700b5f08e-kube-api-access-2tfp9\") pod \"payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5\" (UID: \"003fe297-627b-493b-a55b-cce700b5f08e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"
Sep 07 13:17:33.494357 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:33.494123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003fe297-627b-493b-a55b-cce700b5f08e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5\" (UID: \"003fe297-627b-493b-a55b-cce700b5f08e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"
Sep 07 13:17:33.595412 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:33.595364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tfp9\" (UniqueName: \"kubernetes.io/projected/003fe297-627b-493b-a55b-cce700b5f08e-kube-api-access-2tfp9\") pod \"payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5\" (UID: \"003fe297-627b-493b-a55b-cce700b5f08e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"
Sep 07 13:17:33.595611 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:33.595585    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003fe297-627b-493b-a55b-cce700b5f08e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5\" (UID: \"003fe297-627b-493b-a55b-cce700b5f08e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"
Sep 07 13:17:33.596279 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:33.596258    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003fe297-627b-493b-a55b-cce700b5f08e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5\" (UID: \"003fe297-627b-493b-a55b-cce700b5f08e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"
Sep 07 13:17:33.605417 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:33.605393    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tfp9\" (UniqueName: \"kubernetes.io/projected/003fe297-627b-493b-a55b-cce700b5f08e-kube-api-access-2tfp9\") pod \"payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5\" (UID: \"003fe297-627b-493b-a55b-cce700b5f08e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"
Sep 07 13:17:33.726737 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:33.726638    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"
Sep 07 13:17:34.004569 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:34.004540    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"]
Sep 07 13:17:34.006453 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:17:34.006421    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003fe297_627b_493b_a55b_cce700b5f08e.slice/crio-8252c8540faf50ce231cd6db65eec27afa2ca597ace1b2147475a433c18865d0 WatchSource:0}: Error finding container 8252c8540faf50ce231cd6db65eec27afa2ca597ace1b2147475a433c18865d0: Status 404 returned error can't find the container with id 8252c8540faf50ce231cd6db65eec27afa2ca597ace1b2147475a433c18865d0
Sep 07 13:17:34.286021 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:34.285933    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5" event={"ID":"003fe297-627b-493b-a55b-cce700b5f08e","Type":"ContainerStarted","Data":"e894c8bda353d8f32ba6d163ea8eab2803a055cc74628c4232b7e23daadf6736"}
Sep 07 13:17:34.286021 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:34.285989    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5" event={"ID":"003fe297-627b-493b-a55b-cce700b5f08e","Type":"ContainerStarted","Data":"8252c8540faf50ce231cd6db65eec27afa2ca597ace1b2147475a433c18865d0"}
Sep 07 13:17:34.286021 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:34.286010    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"
Sep 07 13:17:34.308419 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:34.308361    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5" podStartSLOduration=1.308344966 podStartE2EDuration="1.308344966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:17:34.307071467 +0000 UTC m=+1471.337937788" watchObservedRunningTime="2026-09-07 13:17:34.308344966 +0000 UTC m=+1471.339211266"
Sep 07 13:17:42.531321 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:42.531284    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:17:45.294189 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:45.294146    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"
Sep 07 13:17:53.393936 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:53.393821    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c3bc808-8d70-4bbb-9e1f-88a278834552" containerID="bc823883e4d3d8690481ed7d4c5654b2db766a7a02253f254b5e0feb5b8bd31c" exitCode=0
Sep 07 13:17:53.393936 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:53.393897    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g" event={"ID":"3c3bc808-8d70-4bbb-9e1f-88a278834552","Type":"ContainerDied","Data":"bc823883e4d3d8690481ed7d4c5654b2db766a7a02253f254b5e0feb5b8bd31c"}
Sep 07 13:17:53.394465 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:53.394450    2565 scope.go:117] "RemoveContainer" containerID="bc823883e4d3d8690481ed7d4c5654b2db766a7a02253f254b5e0feb5b8bd31c"
Sep 07 13:17:53.395819 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:53.395792    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:17:53.395888 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:53.395865    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:17:53.395929 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:53.395893    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:17:54.406690 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:54.406649    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g" event={"ID":"3c3bc808-8d70-4bbb-9e1f-88a278834552","Type":"ContainerStarted","Data":"a04b3b973ad1105eb1fbcc23a4bde24fbcc3d0ed471981f1f5027d509b169842"}
Sep 07 13:17:54.428377 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:54.428339    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:17:54.428561 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:54.428537    2565 patch_prober.go:28] interesting pod/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 07 13:17:54.428619 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:54.428590    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g" podUID="3c3bc808-8d70-4bbb-9e1f-88a278834552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 07 13:17:55.428585 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:55.428547    2565 patch_prober.go:28] interesting pod/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 07 13:17:55.428938 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:55.428605    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g" podUID="3c3bc808-8d70-4bbb-9e1f-88a278834552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 07 13:17:56.429369 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:56.429307    2565 patch_prober.go:28] interesting pod/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 07 13:17:56.429838 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:56.429391    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g" podUID="3c3bc808-8d70-4bbb-9e1f-88a278834552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 07 13:17:57.428580 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:57.428540    2565 patch_prober.go:28] interesting pod/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 07 13:17:57.428764 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:57.428611    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g" podUID="3c3bc808-8d70-4bbb-9e1f-88a278834552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 07 13:17:58.428937 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:58.428903    2565 patch_prober.go:28] interesting pod/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 07 13:17:58.429369 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:58.428968    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g" podUID="3c3bc808-8d70-4bbb-9e1f-88a278834552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 07 13:17:59.429138 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:59.429096    2565 patch_prober.go:28] interesting pod/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 07 13:17:59.429652 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:17:59.429163    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g" podUID="3c3bc808-8d70-4bbb-9e1f-88a278834552" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 07 13:18:00.260960 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:00.260919    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:18:00.433916 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:00.433882    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:18:00.439919 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:00.439898    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:18:03.680767 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:03.680736    2565 scope.go:117] "RemoveContainer" containerID="6695d2b381ab7e0dc9f60c9fbea7f9e34614162569e780287925e2956a7b3491"
Sep 07 13:18:04.013304 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:04.013215    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:18:18.270839 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:18.270803    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:18:23.954178 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:23.954143    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"]
Sep 07 13:18:24.026514 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:24.026476    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"]
Sep 07 13:18:24.026745 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:24.026720    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z" podUID="991813eb-2fb3-48aa-9908-b5af528eb5f5" containerName="istio-proxy" containerID="cri-o://e1ac81725e61d673fc9e8b843e33b7d16f4aaa2927872a8e200d0158ec0f3b95" gracePeriod=30
Sep 07 13:18:29.284646 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.284618    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:18:29.379258 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.379223    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-envoy\") pod \"991813eb-2fb3-48aa-9908-b5af528eb5f5\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") "
Sep 07 13:18:29.379462 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.379321    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-credential-socket\") pod \"991813eb-2fb3-48aa-9908-b5af528eb5f5\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") "
Sep 07 13:18:29.379462 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.379379    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxv6z\" (UniqueName: \"kubernetes.io/projected/991813eb-2fb3-48aa-9908-b5af528eb5f5-kube-api-access-wxv6z\") pod \"991813eb-2fb3-48aa-9908-b5af528eb5f5\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") "
Sep 07 13:18:29.379462 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.379403    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-token\") pod \"991813eb-2fb3-48aa-9908-b5af528eb5f5\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") "
Sep 07 13:18:29.379462 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.379431    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-podinfo\") pod \"991813eb-2fb3-48aa-9908-b5af528eb5f5\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") "
Sep 07 13:18:29.379462 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.379458    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991813eb-2fb3-48aa-9908-b5af528eb5f5-istiod-ca-cert\") pod \"991813eb-2fb3-48aa-9908-b5af528eb5f5\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") "
Sep 07 13:18:29.379724 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.379500    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-data\") pod \"991813eb-2fb3-48aa-9908-b5af528eb5f5\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") "
Sep 07 13:18:29.379724 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.379563    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-workload-certs\") pod \"991813eb-2fb3-48aa-9908-b5af528eb5f5\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") "
Sep 07 13:18:29.379724 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.379593    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-workload-socket\") pod \"991813eb-2fb3-48aa-9908-b5af528eb5f5\" (UID: \"991813eb-2fb3-48aa-9908-b5af528eb5f5\") "
Sep 07 13:18:29.379724 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.379674    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "991813eb-2fb3-48aa-9908-b5af528eb5f5" (UID: "991813eb-2fb3-48aa-9908-b5af528eb5f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:29.379943 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.379890    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "991813eb-2fb3-48aa-9908-b5af528eb5f5" (UID: "991813eb-2fb3-48aa-9908-b5af528eb5f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:29.379997 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.379939    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991813eb-2fb3-48aa-9908-b5af528eb5f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "991813eb-2fb3-48aa-9908-b5af528eb5f5" (UID: "991813eb-2fb3-48aa-9908-b5af528eb5f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:18:29.380195 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.380100    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "991813eb-2fb3-48aa-9908-b5af528eb5f5" (UID: "991813eb-2fb3-48aa-9908-b5af528eb5f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:29.380195 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.380131    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-credential-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.380195 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.380153    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/991813eb-2fb3-48aa-9908-b5af528eb5f5-istiod-ca-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.380195 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.380167    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-data\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.380571 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.380411    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "991813eb-2fb3-48aa-9908-b5af528eb5f5" (UID: "991813eb-2fb3-48aa-9908-b5af528eb5f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:29.381692 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.381668    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "991813eb-2fb3-48aa-9908-b5af528eb5f5" (UID: "991813eb-2fb3-48aa-9908-b5af528eb5f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:29.381861 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.381843    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "991813eb-2fb3-48aa-9908-b5af528eb5f5" (UID: "991813eb-2fb3-48aa-9908-b5af528eb5f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:18:29.381924 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.381866    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "991813eb-2fb3-48aa-9908-b5af528eb5f5" (UID: "991813eb-2fb3-48aa-9908-b5af528eb5f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 13:18:29.381924 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.381883    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991813eb-2fb3-48aa-9908-b5af528eb5f5-kube-api-access-wxv6z" (OuterVolumeSpecName: "kube-api-access-wxv6z") pod "991813eb-2fb3-48aa-9908-b5af528eb5f5" (UID: "991813eb-2fb3-48aa-9908-b5af528eb5f5"). InnerVolumeSpecName "kube-api-access-wxv6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:18:29.481591 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.481546    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-workload-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.481591 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.481588    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-workload-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.481591 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.481599    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-envoy\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.481591 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.481609    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxv6z\" (UniqueName: \"kubernetes.io/projected/991813eb-2fb3-48aa-9908-b5af528eb5f5-kube-api-access-wxv6z\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.481864 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.481619    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-token\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.481864 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.481627    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/991813eb-2fb3-48aa-9908-b5af528eb5f5-istio-podinfo\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:29.616791 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.616752    2565 generic.go:356] "Generic (PLEG): container finished" podID="991813eb-2fb3-48aa-9908-b5af528eb5f5" containerID="e1ac81725e61d673fc9e8b843e33b7d16f4aaa2927872a8e200d0158ec0f3b95" exitCode=0
Sep 07 13:18:29.616953 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.616803    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"
Sep 07 13:18:29.616953 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.616801    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z" event={"ID":"991813eb-2fb3-48aa-9908-b5af528eb5f5","Type":"ContainerDied","Data":"e1ac81725e61d673fc9e8b843e33b7d16f4aaa2927872a8e200d0158ec0f3b95"}
Sep 07 13:18:29.616953 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.616918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z" event={"ID":"991813eb-2fb3-48aa-9908-b5af528eb5f5","Type":"ContainerDied","Data":"b64857104c542f654892f7c8788a742f461123b3f69849e4d24437c63c77d06d"}
Sep 07 13:18:29.616953 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.616947    2565 scope.go:117] "RemoveContainer" containerID="e1ac81725e61d673fc9e8b843e33b7d16f4aaa2927872a8e200d0158ec0f3b95"
Sep 07 13:18:29.630065 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.630042    2565 scope.go:117] "RemoveContainer" containerID="e6a50a37ad0d414d8758bc08867c59d04a5295c50770b2d0d84c66f76fd88eaa"
Sep 07 13:18:29.639074 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.639048    2565 scope.go:117] "RemoveContainer" containerID="e1ac81725e61d673fc9e8b843e33b7d16f4aaa2927872a8e200d0158ec0f3b95"
Sep 07 13:18:29.639378 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:18:29.639348    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ac81725e61d673fc9e8b843e33b7d16f4aaa2927872a8e200d0158ec0f3b95\": container with ID starting with e1ac81725e61d673fc9e8b843e33b7d16f4aaa2927872a8e200d0158ec0f3b95 not found: ID does not exist" containerID="e1ac81725e61d673fc9e8b843e33b7d16f4aaa2927872a8e200d0158ec0f3b95"
Sep 07 13:18:29.639480 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.639389    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ac81725e61d673fc9e8b843e33b7d16f4aaa2927872a8e200d0158ec0f3b95"} err="failed to get container status \"e1ac81725e61d673fc9e8b843e33b7d16f4aaa2927872a8e200d0158ec0f3b95\": rpc error: code = NotFound desc = could not find container \"e1ac81725e61d673fc9e8b843e33b7d16f4aaa2927872a8e200d0158ec0f3b95\": container with ID starting with e1ac81725e61d673fc9e8b843e33b7d16f4aaa2927872a8e200d0158ec0f3b95 not found: ID does not exist"
Sep 07 13:18:29.639480 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.639407    2565 scope.go:117] "RemoveContainer" containerID="e6a50a37ad0d414d8758bc08867c59d04a5295c50770b2d0d84c66f76fd88eaa"
Sep 07 13:18:29.639708 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:18:29.639694    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6a50a37ad0d414d8758bc08867c59d04a5295c50770b2d0d84c66f76fd88eaa\": container with ID starting with e6a50a37ad0d414d8758bc08867c59d04a5295c50770b2d0d84c66f76fd88eaa not found: ID does not exist" containerID="e6a50a37ad0d414d8758bc08867c59d04a5295c50770b2d0d84c66f76fd88eaa"
Sep 07 13:18:29.639756 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.639713    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6a50a37ad0d414d8758bc08867c59d04a5295c50770b2d0d84c66f76fd88eaa"} err="failed to get container status \"e6a50a37ad0d414d8758bc08867c59d04a5295c50770b2d0d84c66f76fd88eaa\": rpc error: code = NotFound desc = could not find container \"e6a50a37ad0d414d8758bc08867c59d04a5295c50770b2d0d84c66f76fd88eaa\": container with ID starting with e6a50a37ad0d414d8758bc08867c59d04a5295c50770b2d0d84c66f76fd88eaa not found: ID does not exist"
Sep 07 13:18:29.651595 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.651569    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"]
Sep 07 13:18:29.653725 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:29.653704    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ddb9ca-openshift-default-bf58f6ff4-cmr8z"]
Sep 07 13:18:31.177757 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.177723    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"]
Sep 07 13:18:31.178213 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.178198    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991813eb-2fb3-48aa-9908-b5af528eb5f5" containerName="istio-proxy"
Sep 07 13:18:31.178258 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.178214    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="991813eb-2fb3-48aa-9908-b5af528eb5f5" containerName="istio-proxy"
Sep 07 13:18:31.178293 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.178286    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="991813eb-2fb3-48aa-9908-b5af528eb5f5" containerName="istio-proxy"
Sep 07 13:18:31.178361 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.178295    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="991813eb-2fb3-48aa-9908-b5af528eb5f5" containerName="istio-proxy"
Sep 07 13:18:31.183119 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.183101    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"
Sep 07 13:18:31.185559 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.185539    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ec10ff-dockercfg-5cb84\""
Sep 07 13:18:31.185704 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.185601    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ec10ff\""
Sep 07 13:18:31.194992 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.194967    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"]
Sep 07 13:18:31.274064 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.274025    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"]
Sep 07 13:18:31.274579 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.274561    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991813eb-2fb3-48aa-9908-b5af528eb5f5" containerName="istio-proxy"
Sep 07 13:18:31.274579 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.274580    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="991813eb-2fb3-48aa-9908-b5af528eb5f5" containerName="istio-proxy"
Sep 07 13:18:31.277914 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.277888    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"
Sep 07 13:18:31.292440 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.292404    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"]
Sep 07 13:18:31.300130 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.300101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18eb88e-3691-43bc-adb0-c0bd0f535a35-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c\" (UID: \"f18eb88e-3691-43bc-adb0-c0bd0f535a35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"
Sep 07 13:18:31.300370 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.300315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvz5w\" (UniqueName: \"kubernetes.io/projected/f18eb88e-3691-43bc-adb0-c0bd0f535a35-kube-api-access-xvz5w\") pod \"payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c\" (UID: \"f18eb88e-3691-43bc-adb0-c0bd0f535a35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"
Sep 07 13:18:31.401111 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.401069    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bc72f1-b2f2-4221-98aa-50e9af10db28-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw\" (UID: \"52bc72f1-b2f2-4221-98aa-50e9af10db28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"
Sep 07 13:18:31.401302 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.401125    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kzxg\" (UniqueName: \"kubernetes.io/projected/52bc72f1-b2f2-4221-98aa-50e9af10db28-kube-api-access-8kzxg\") pod \"payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw\" (UID: \"52bc72f1-b2f2-4221-98aa-50e9af10db28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"
Sep 07 13:18:31.401302 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.401230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvz5w\" (UniqueName: \"kubernetes.io/projected/f18eb88e-3691-43bc-adb0-c0bd0f535a35-kube-api-access-xvz5w\") pod \"payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c\" (UID: \"f18eb88e-3691-43bc-adb0-c0bd0f535a35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"
Sep 07 13:18:31.401302 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.401286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18eb88e-3691-43bc-adb0-c0bd0f535a35-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c\" (UID: \"f18eb88e-3691-43bc-adb0-c0bd0f535a35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"
Sep 07 13:18:31.402024 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.401998    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18eb88e-3691-43bc-adb0-c0bd0f535a35-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c\" (UID: \"f18eb88e-3691-43bc-adb0-c0bd0f535a35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"
Sep 07 13:18:31.412525 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.412497    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvz5w\" (UniqueName: \"kubernetes.io/projected/f18eb88e-3691-43bc-adb0-c0bd0f535a35-kube-api-access-xvz5w\") pod \"payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c\" (UID: \"f18eb88e-3691-43bc-adb0-c0bd0f535a35\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"
Sep 07 13:18:31.488823 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.488748    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="991813eb-2fb3-48aa-9908-b5af528eb5f5" path="/var/lib/kubelet/pods/991813eb-2fb3-48aa-9908-b5af528eb5f5/volumes"
Sep 07 13:18:31.495060 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.495039    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"
Sep 07 13:18:31.502512 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.502485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bc72f1-b2f2-4221-98aa-50e9af10db28-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw\" (UID: \"52bc72f1-b2f2-4221-98aa-50e9af10db28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"
Sep 07 13:18:31.502610 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.502528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kzxg\" (UniqueName: \"kubernetes.io/projected/52bc72f1-b2f2-4221-98aa-50e9af10db28-kube-api-access-8kzxg\") pod \"payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw\" (UID: \"52bc72f1-b2f2-4221-98aa-50e9af10db28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"
Sep 07 13:18:31.503087 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.503070    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bc72f1-b2f2-4221-98aa-50e9af10db28-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw\" (UID: \"52bc72f1-b2f2-4221-98aa-50e9af10db28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"
Sep 07 13:18:31.513149 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.513125    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kzxg\" (UniqueName: \"kubernetes.io/projected/52bc72f1-b2f2-4221-98aa-50e9af10db28-kube-api-access-8kzxg\") pod \"payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw\" (UID: \"52bc72f1-b2f2-4221-98aa-50e9af10db28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"
Sep 07 13:18:31.592859 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.592821    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"
Sep 07 13:18:31.774988 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:18:31.774959    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18eb88e_3691_43bc_adb0_c0bd0f535a35.slice/crio-290a6c5eda794a9e103c665cdd2b6fbea8b31a4a84eb0312c7a0387f2df6e383 WatchSource:0}: Error finding container 290a6c5eda794a9e103c665cdd2b6fbea8b31a4a84eb0312c7a0387f2df6e383: Status 404 returned error can't find the container with id 290a6c5eda794a9e103c665cdd2b6fbea8b31a4a84eb0312c7a0387f2df6e383
Sep 07 13:18:31.775145 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.775120    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"]
Sep 07 13:18:31.873068 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:31.873037    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"]
Sep 07 13:18:31.875128 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:18:31.875097    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bc72f1_b2f2_4221_98aa_50e9af10db28.slice/crio-7cf7c270303d06701585958a7fda9e34ee2905a83ee589ecf184ddd103d742bf WatchSource:0}: Error finding container 7cf7c270303d06701585958a7fda9e34ee2905a83ee589ecf184ddd103d742bf: Status 404 returned error can't find the container with id 7cf7c270303d06701585958a7fda9e34ee2905a83ee589ecf184ddd103d742bf
Sep 07 13:18:32.113957 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:32.113915    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:18:32.635685 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:32.635652    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c" event={"ID":"f18eb88e-3691-43bc-adb0-c0bd0f535a35","Type":"ContainerStarted","Data":"049032e15e91ba8503987265f19dc96fab32e754aeb38f3d0171331dc115b82e"}
Sep 07 13:18:32.636163 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:32.635710    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c" event={"ID":"f18eb88e-3691-43bc-adb0-c0bd0f535a35","Type":"ContainerStarted","Data":"290a6c5eda794a9e103c665cdd2b6fbea8b31a4a84eb0312c7a0387f2df6e383"}
Sep 07 13:18:32.636163 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:32.635840    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"
Sep 07 13:18:32.637516 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:32.637488    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw" event={"ID":"52bc72f1-b2f2-4221-98aa-50e9af10db28","Type":"ContainerStarted","Data":"4137c757fad7e02bca56884d30c88af9ad6cc9ef10eda2a5943e5ecc74c0c5dd"}
Sep 07 13:18:32.637645 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:32.637531    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw" event={"ID":"52bc72f1-b2f2-4221-98aa-50e9af10db28","Type":"ContainerStarted","Data":"7cf7c270303d06701585958a7fda9e34ee2905a83ee589ecf184ddd103d742bf"}
Sep 07 13:18:32.637645 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:32.637566    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"
Sep 07 13:18:32.654825 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:32.654773    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c" podStartSLOduration=1.654754026 podStartE2EDuration="1.654754026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:18:32.653986132 +0000 UTC m=+1529.684852442" watchObservedRunningTime="2026-09-07 13:18:32.654754026 +0000 UTC m=+1529.685620338"
Sep 07 13:18:32.679177 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:32.679118    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw" podStartSLOduration=1.679101671 podStartE2EDuration="1.679101671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:18:32.676802644 +0000 UTC m=+1529.707668955" watchObservedRunningTime="2026-09-07 13:18:32.679101671 +0000 UTC m=+1529.709967981"
Sep 07 13:18:38.974991 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:38.974949    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8" podUID="641b1958-7356-4bd2-add2-a7a7afaa8713" containerName="main" containerID="cri-o://68b895313119f34072a5fe37a28d591dc2c2892fc4fd95aba1e82981d3ef5d7a" gracePeriod=45
Sep 07 13:18:39.238227 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.238163    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:18:39.383850 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.383806    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-tmp-dir\") pod \"641b1958-7356-4bd2-add2-a7a7afaa8713\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") "
Sep 07 13:18:39.384025 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.383870    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-model-cache\") pod \"641b1958-7356-4bd2-add2-a7a7afaa8713\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") "
Sep 07 13:18:39.384025 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.383913    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-home\") pod \"641b1958-7356-4bd2-add2-a7a7afaa8713\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") "
Sep 07 13:18:39.384025 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.383959    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-dshm\") pod \"641b1958-7356-4bd2-add2-a7a7afaa8713\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") "
Sep 07 13:18:39.384144 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.384029    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641b1958-7356-4bd2-add2-a7a7afaa8713-tls-certs\") pod \"641b1958-7356-4bd2-add2-a7a7afaa8713\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") "
Sep 07 13:18:39.384144 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.384085    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvbf4\" (UniqueName: \"kubernetes.io/projected/641b1958-7356-4bd2-add2-a7a7afaa8713-kube-api-access-lvbf4\") pod \"641b1958-7356-4bd2-add2-a7a7afaa8713\" (UID: \"641b1958-7356-4bd2-add2-a7a7afaa8713\") "
Sep 07 13:18:39.384259 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.384153    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "641b1958-7356-4bd2-add2-a7a7afaa8713" (UID: "641b1958-7356-4bd2-add2-a7a7afaa8713"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:39.384259 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.384236    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-model-cache" (OuterVolumeSpecName: "model-cache") pod "641b1958-7356-4bd2-add2-a7a7afaa8713" (UID: "641b1958-7356-4bd2-add2-a7a7afaa8713"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:39.384395 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.384255    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-home" (OuterVolumeSpecName: "home") pod "641b1958-7356-4bd2-add2-a7a7afaa8713" (UID: "641b1958-7356-4bd2-add2-a7a7afaa8713"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:39.384516 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.384497    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-tmp-dir\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:39.384567 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.384524    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-model-cache\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:39.384567 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.384539    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-home\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:39.386279 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.386253    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641b1958-7356-4bd2-add2-a7a7afaa8713-kube-api-access-lvbf4" (OuterVolumeSpecName: "kube-api-access-lvbf4") pod "641b1958-7356-4bd2-add2-a7a7afaa8713" (UID: "641b1958-7356-4bd2-add2-a7a7afaa8713"). InnerVolumeSpecName "kube-api-access-lvbf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:18:39.386462 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.386277    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-dshm" (OuterVolumeSpecName: "dshm") pod "641b1958-7356-4bd2-add2-a7a7afaa8713" (UID: "641b1958-7356-4bd2-add2-a7a7afaa8713"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:39.386462 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.386276    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641b1958-7356-4bd2-add2-a7a7afaa8713-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "641b1958-7356-4bd2-add2-a7a7afaa8713" (UID: "641b1958-7356-4bd2-add2-a7a7afaa8713"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:18:39.485071 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.485041    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641b1958-7356-4bd2-add2-a7a7afaa8713-dshm\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:39.485071 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.485069    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641b1958-7356-4bd2-add2-a7a7afaa8713-tls-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:39.485071 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.485078    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvbf4\" (UniqueName: \"kubernetes.io/projected/641b1958-7356-4bd2-add2-a7a7afaa8713-kube-api-access-lvbf4\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:39.681164 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.681129    2565 generic.go:356] "Generic (PLEG): container finished" podID="641b1958-7356-4bd2-add2-a7a7afaa8713" containerID="68b895313119f34072a5fe37a28d591dc2c2892fc4fd95aba1e82981d3ef5d7a" exitCode=0
Sep 07 13:18:39.681392 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.681171    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"
Sep 07 13:18:39.681392 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.681214    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8" event={"ID":"641b1958-7356-4bd2-add2-a7a7afaa8713","Type":"ContainerDied","Data":"68b895313119f34072a5fe37a28d591dc2c2892fc4fd95aba1e82981d3ef5d7a"}
Sep 07 13:18:39.681392 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.681263    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8" event={"ID":"641b1958-7356-4bd2-add2-a7a7afaa8713","Type":"ContainerDied","Data":"23f903e5f53d74d809dabb324139c5b5807090e1292fbd2bd810f00aaf91bd07"}
Sep 07 13:18:39.681392 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.681284    2565 scope.go:117] "RemoveContainer" containerID="68b895313119f34072a5fe37a28d591dc2c2892fc4fd95aba1e82981d3ef5d7a"
Sep 07 13:18:39.690415 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.690396    2565 scope.go:117] "RemoveContainer" containerID="68b895313119f34072a5fe37a28d591dc2c2892fc4fd95aba1e82981d3ef5d7a"
Sep 07 13:18:39.690690 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:18:39.690666    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b895313119f34072a5fe37a28d591dc2c2892fc4fd95aba1e82981d3ef5d7a\": container with ID starting with 68b895313119f34072a5fe37a28d591dc2c2892fc4fd95aba1e82981d3ef5d7a not found: ID does not exist" containerID="68b895313119f34072a5fe37a28d591dc2c2892fc4fd95aba1e82981d3ef5d7a"
Sep 07 13:18:39.690752 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.690700    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b895313119f34072a5fe37a28d591dc2c2892fc4fd95aba1e82981d3ef5d7a"} err="failed to get container status \"68b895313119f34072a5fe37a28d591dc2c2892fc4fd95aba1e82981d3ef5d7a\": rpc error: code = NotFound desc = could not find container \"68b895313119f34072a5fe37a28d591dc2c2892fc4fd95aba1e82981d3ef5d7a\": container with ID starting with 68b895313119f34072a5fe37a28d591dc2c2892fc4fd95aba1e82981d3ef5d7a not found: ID does not exist"
Sep 07 13:18:39.724938 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.724870    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"]
Sep 07 13:18:39.727516 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:39.727494    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aa813e/rate-test-model-aa813e-kserve-7cc54cc847-5zds8"]
Sep 07 13:18:40.587012 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:40.586980    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"]
Sep 07 13:18:40.587477 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:40.587193    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5" podUID="003fe297-627b-493b-a55b-cce700b5f08e" containerName="payload-pre-processing" containerID="cri-o://e894c8bda353d8f32ba6d163ea8eab2803a055cc74628c4232b7e23daadf6736" gracePeriod=30
Sep 07 13:18:40.855664 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:40.855637    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"
Sep 07 13:18:41.001532 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:41.001494    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003fe297-627b-493b-a55b-cce700b5f08e-plugins-config-volume\") pod \"003fe297-627b-493b-a55b-cce700b5f08e\" (UID: \"003fe297-627b-493b-a55b-cce700b5f08e\") "
Sep 07 13:18:41.001697 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:41.001573    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tfp9\" (UniqueName: \"kubernetes.io/projected/003fe297-627b-493b-a55b-cce700b5f08e-kube-api-access-2tfp9\") pod \"003fe297-627b-493b-a55b-cce700b5f08e\" (UID: \"003fe297-627b-493b-a55b-cce700b5f08e\") "
Sep 07 13:18:41.001894 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:41.001871    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003fe297-627b-493b-a55b-cce700b5f08e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "003fe297-627b-493b-a55b-cce700b5f08e" (UID: "003fe297-627b-493b-a55b-cce700b5f08e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:18:41.003737 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:41.003713    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003fe297-627b-493b-a55b-cce700b5f08e-kube-api-access-2tfp9" (OuterVolumeSpecName: "kube-api-access-2tfp9") pod "003fe297-627b-493b-a55b-cce700b5f08e" (UID: "003fe297-627b-493b-a55b-cce700b5f08e"). InnerVolumeSpecName "kube-api-access-2tfp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:18:41.103233 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:41.103137    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/003fe297-627b-493b-a55b-cce700b5f08e-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:41.103233 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:41.103170    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tfp9\" (UniqueName: \"kubernetes.io/projected/003fe297-627b-493b-a55b-cce700b5f08e-kube-api-access-2tfp9\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:41.488992 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:41.488908    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="641b1958-7356-4bd2-add2-a7a7afaa8713" path="/var/lib/kubelet/pods/641b1958-7356-4bd2-add2-a7a7afaa8713/volumes"
Sep 07 13:18:41.695215 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:41.695173    2565 generic.go:356] "Generic (PLEG): container finished" podID="003fe297-627b-493b-a55b-cce700b5f08e" containerID="e894c8bda353d8f32ba6d163ea8eab2803a055cc74628c4232b7e23daadf6736" exitCode=0
Sep 07 13:18:41.695758 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:41.695222    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"
Sep 07 13:18:41.695758 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:41.695254    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5" event={"ID":"003fe297-627b-493b-a55b-cce700b5f08e","Type":"ContainerDied","Data":"e894c8bda353d8f32ba6d163ea8eab2803a055cc74628c4232b7e23daadf6736"}
Sep 07 13:18:41.695758 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:41.695310    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5" event={"ID":"003fe297-627b-493b-a55b-cce700b5f08e","Type":"ContainerDied","Data":"8252c8540faf50ce231cd6db65eec27afa2ca597ace1b2147475a433c18865d0"}
Sep 07 13:18:41.695758 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:41.695345    2565 scope.go:117] "RemoveContainer" containerID="e894c8bda353d8f32ba6d163ea8eab2803a055cc74628c4232b7e23daadf6736"
Sep 07 13:18:41.704906 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:41.704885    2565 scope.go:117] "RemoveContainer" containerID="e894c8bda353d8f32ba6d163ea8eab2803a055cc74628c4232b7e23daadf6736"
Sep 07 13:18:41.705180 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:18:41.705161    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e894c8bda353d8f32ba6d163ea8eab2803a055cc74628c4232b7e23daadf6736\": container with ID starting with e894c8bda353d8f32ba6d163ea8eab2803a055cc74628c4232b7e23daadf6736 not found: ID does not exist" containerID="e894c8bda353d8f32ba6d163ea8eab2803a055cc74628c4232b7e23daadf6736"
Sep 07 13:18:41.705228 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:41.705188    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e894c8bda353d8f32ba6d163ea8eab2803a055cc74628c4232b7e23daadf6736"} err="failed to get container status \"e894c8bda353d8f32ba6d163ea8eab2803a055cc74628c4232b7e23daadf6736\": rpc error: code = NotFound desc = could not find container \"e894c8bda353d8f32ba6d163ea8eab2803a055cc74628c4232b7e23daadf6736\": container with ID starting with e894c8bda353d8f32ba6d163ea8eab2803a055cc74628c4232b7e23daadf6736 not found: ID does not exist"
Sep 07 13:18:41.715837 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:41.715804    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"]
Sep 07 13:18:41.720609 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:41.720580    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0d2b4b-584bfbbdf8-9g7t5"]
Sep 07 13:18:42.313854 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:42.313817    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:18:43.489792 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:43.489757    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="003fe297-627b-493b-a55b-cce700b5f08e" path="/var/lib/kubelet/pods/003fe297-627b-493b-a55b-cce700b5f08e/volumes"
Sep 07 13:18:43.645550 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:43.645520    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"
Sep 07 13:18:43.645710 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:43.645676    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"
Sep 07 13:18:48.651266 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:48.651232    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"]
Sep 07 13:18:48.651729 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:48.651496    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g" podUID="3c3bc808-8d70-4bbb-9e1f-88a278834552" containerName="istio-proxy" containerID="cri-o://a04b3b973ad1105eb1fbcc23a4bde24fbcc3d0ed471981f1f5027d509b169842" gracePeriod=30
Sep 07 13:18:53.917449 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.917381    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:18:53.937063 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937036    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-data\") pod \"3c3bc808-8d70-4bbb-9e1f-88a278834552\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") "
Sep 07 13:18:53.937233 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937093    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwckm\" (UniqueName: \"kubernetes.io/projected/3c3bc808-8d70-4bbb-9e1f-88a278834552-kube-api-access-cwckm\") pod \"3c3bc808-8d70-4bbb-9e1f-88a278834552\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") "
Sep 07 13:18:53.937233 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937123    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-credential-socket\") pod \"3c3bc808-8d70-4bbb-9e1f-88a278834552\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") "
Sep 07 13:18:53.937233 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937137    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-token\") pod \"3c3bc808-8d70-4bbb-9e1f-88a278834552\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") "
Sep 07 13:18:53.937233 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937152    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-workload-certs\") pod \"3c3bc808-8d70-4bbb-9e1f-88a278834552\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") "
Sep 07 13:18:53.937233 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937179    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-workload-socket\") pod \"3c3bc808-8d70-4bbb-9e1f-88a278834552\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") "
Sep 07 13:18:53.937233 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937222    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-envoy\") pod \"3c3bc808-8d70-4bbb-9e1f-88a278834552\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") "
Sep 07 13:18:53.937577 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937245    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3bc808-8d70-4bbb-9e1f-88a278834552-istiod-ca-cert\") pod \"3c3bc808-8d70-4bbb-9e1f-88a278834552\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") "
Sep 07 13:18:53.937577 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937298    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-podinfo\") pod \"3c3bc808-8d70-4bbb-9e1f-88a278834552\" (UID: \"3c3bc808-8d70-4bbb-9e1f-88a278834552\") "
Sep 07 13:18:53.937577 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937374    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c3bc808-8d70-4bbb-9e1f-88a278834552" (UID: "3c3bc808-8d70-4bbb-9e1f-88a278834552"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:53.937577 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937435    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c3bc808-8d70-4bbb-9e1f-88a278834552" (UID: "3c3bc808-8d70-4bbb-9e1f-88a278834552"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:53.937794 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937618    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c3bc808-8d70-4bbb-9e1f-88a278834552" (UID: "3c3bc808-8d70-4bbb-9e1f-88a278834552"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:53.937794 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937740    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3bc808-8d70-4bbb-9e1f-88a278834552-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c3bc808-8d70-4bbb-9e1f-88a278834552" (UID: "3c3bc808-8d70-4bbb-9e1f-88a278834552"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:18:53.937794 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937768    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-data\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:53.937794 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937789    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-credential-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:53.937943 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937803    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-workload-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:53.937943 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.937806    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c3bc808-8d70-4bbb-9e1f-88a278834552" (UID: "3c3bc808-8d70-4bbb-9e1f-88a278834552"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:53.939478 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.939441    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c3bc808-8d70-4bbb-9e1f-88a278834552" (UID: "3c3bc808-8d70-4bbb-9e1f-88a278834552"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:18:53.939865 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.939845    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c3bc808-8d70-4bbb-9e1f-88a278834552" (UID: "3c3bc808-8d70-4bbb-9e1f-88a278834552"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 13:18:53.939977 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.939928    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c3bc808-8d70-4bbb-9e1f-88a278834552" (UID: "3c3bc808-8d70-4bbb-9e1f-88a278834552"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:18:53.939977 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:53.939946    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3bc808-8d70-4bbb-9e1f-88a278834552-kube-api-access-cwckm" (OuterVolumeSpecName: "kube-api-access-cwckm") pod "3c3bc808-8d70-4bbb-9e1f-88a278834552" (UID: "3c3bc808-8d70-4bbb-9e1f-88a278834552"). InnerVolumeSpecName "kube-api-access-cwckm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:18:54.038756 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.038721    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwckm\" (UniqueName: \"kubernetes.io/projected/3c3bc808-8d70-4bbb-9e1f-88a278834552-kube-api-access-cwckm\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:54.038756 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.038753    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-token\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:54.038756 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.038763    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-workload-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:54.039032 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.038770    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-envoy\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:54.039032 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.038780    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3bc808-8d70-4bbb-9e1f-88a278834552-istiod-ca-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:54.039032 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.038787    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3bc808-8d70-4bbb-9e1f-88a278834552-istio-podinfo\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:18:54.774034 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.773996    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c3bc808-8d70-4bbb-9e1f-88a278834552" containerID="a04b3b973ad1105eb1fbcc23a4bde24fbcc3d0ed471981f1f5027d509b169842" exitCode=0
Sep 07 13:18:54.774228 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.774045    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"
Sep 07 13:18:54.774228 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.774079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g" event={"ID":"3c3bc808-8d70-4bbb-9e1f-88a278834552","Type":"ContainerDied","Data":"a04b3b973ad1105eb1fbcc23a4bde24fbcc3d0ed471981f1f5027d509b169842"}
Sep 07 13:18:54.774228 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.774137    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g" event={"ID":"3c3bc808-8d70-4bbb-9e1f-88a278834552","Type":"ContainerDied","Data":"64682800bca7986143d10f1f9892ee64695ef2a9b795621f358f72f47c155b86"}
Sep 07 13:18:54.774228 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.774169    2565 scope.go:117] "RemoveContainer" containerID="a04b3b973ad1105eb1fbcc23a4bde24fbcc3d0ed471981f1f5027d509b169842"
Sep 07 13:18:54.784652 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.784636    2565 scope.go:117] "RemoveContainer" containerID="bc823883e4d3d8690481ed7d4c5654b2db766a7a02253f254b5e0feb5b8bd31c"
Sep 07 13:18:54.793412 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.793392    2565 scope.go:117] "RemoveContainer" containerID="a04b3b973ad1105eb1fbcc23a4bde24fbcc3d0ed471981f1f5027d509b169842"
Sep 07 13:18:54.793693 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:18:54.793673    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a04b3b973ad1105eb1fbcc23a4bde24fbcc3d0ed471981f1f5027d509b169842\": container with ID starting with a04b3b973ad1105eb1fbcc23a4bde24fbcc3d0ed471981f1f5027d509b169842 not found: ID does not exist" containerID="a04b3b973ad1105eb1fbcc23a4bde24fbcc3d0ed471981f1f5027d509b169842"
Sep 07 13:18:54.793773 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.793701    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a04b3b973ad1105eb1fbcc23a4bde24fbcc3d0ed471981f1f5027d509b169842"} err="failed to get container status \"a04b3b973ad1105eb1fbcc23a4bde24fbcc3d0ed471981f1f5027d509b169842\": rpc error: code = NotFound desc = could not find container \"a04b3b973ad1105eb1fbcc23a4bde24fbcc3d0ed471981f1f5027d509b169842\": container with ID starting with a04b3b973ad1105eb1fbcc23a4bde24fbcc3d0ed471981f1f5027d509b169842 not found: ID does not exist"
Sep 07 13:18:54.793773 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.793718    2565 scope.go:117] "RemoveContainer" containerID="bc823883e4d3d8690481ed7d4c5654b2db766a7a02253f254b5e0feb5b8bd31c"
Sep 07 13:18:54.793971 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:18:54.793953    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc823883e4d3d8690481ed7d4c5654b2db766a7a02253f254b5e0feb5b8bd31c\": container with ID starting with bc823883e4d3d8690481ed7d4c5654b2db766a7a02253f254b5e0feb5b8bd31c not found: ID does not exist" containerID="bc823883e4d3d8690481ed7d4c5654b2db766a7a02253f254b5e0feb5b8bd31c"
Sep 07 13:18:54.794014 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.793977    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc823883e4d3d8690481ed7d4c5654b2db766a7a02253f254b5e0feb5b8bd31c"} err="failed to get container status \"bc823883e4d3d8690481ed7d4c5654b2db766a7a02253f254b5e0feb5b8bd31c\": rpc error: code = NotFound desc = could not find container \"bc823883e4d3d8690481ed7d4c5654b2db766a7a02253f254b5e0feb5b8bd31c\": container with ID starting with bc823883e4d3d8690481ed7d4c5654b2db766a7a02253f254b5e0feb5b8bd31c not found: ID does not exist"
Sep 07 13:18:54.801735 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.801711    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"]
Sep 07 13:18:54.806967 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:54.806945    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0d2b4b-openshift-default-848845c6db-h5v7g"]
Sep 07 13:18:55.456753 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.456712    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"]
Sep 07 13:18:55.457283 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.457156    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3bc808-8d70-4bbb-9e1f-88a278834552" containerName="istio-proxy"
Sep 07 13:18:55.457283 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.457170    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3bc808-8d70-4bbb-9e1f-88a278834552" containerName="istio-proxy"
Sep 07 13:18:55.457283 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.457180    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003fe297-627b-493b-a55b-cce700b5f08e" containerName="payload-pre-processing"
Sep 07 13:18:55.457283 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.457189    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="003fe297-627b-493b-a55b-cce700b5f08e" containerName="payload-pre-processing"
Sep 07 13:18:55.457283 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.457223    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="641b1958-7356-4bd2-add2-a7a7afaa8713" containerName="main"
Sep 07 13:18:55.457283 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.457229    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="641b1958-7356-4bd2-add2-a7a7afaa8713" containerName="main"
Sep 07 13:18:55.457531 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.457300    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="003fe297-627b-493b-a55b-cce700b5f08e" containerName="payload-pre-processing"
Sep 07 13:18:55.457531 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.457308    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3bc808-8d70-4bbb-9e1f-88a278834552" containerName="istio-proxy"
Sep 07 13:18:55.457531 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.457316    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="641b1958-7356-4bd2-add2-a7a7afaa8713" containerName="main"
Sep 07 13:18:55.470491 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.470463    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.475054 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.475022    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-688b44-openshift-default-dockercfg-9ml7w\""
Sep 07 13:18:55.477866 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.477841    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"]
Sep 07 13:18:55.489909 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.489873    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c3bc808-8d70-4bbb-9e1f-88a278834552" path="/var/lib/kubelet/pods/3c3bc808-8d70-4bbb-9e1f-88a278834552/volumes"
Sep 07 13:18:55.557566 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.555692    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-workload-certs\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.557566 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.555831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-envoy\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.557566 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.555898    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-credential-socket\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.557566 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.555929    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-workload-socket\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.557566 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.555963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-token\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.557566 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.556032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87ft8\" (UniqueName: \"kubernetes.io/projected/bd4cbe7f-35b4-45df-92d7-778847de0c25-kube-api-access-87ft8\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.557566 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.556063    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd4cbe7f-35b4-45df-92d7-778847de0c25-istiod-ca-cert\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.557566 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.556159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-data\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.557566 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.556198    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-podinfo\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.656987 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.656952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-envoy\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.657191 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.656994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-credential-socket\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.657191 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.657022    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-workload-socket\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.657191 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.657052    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-token\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.657191 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.657089    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87ft8\" (UniqueName: \"kubernetes.io/projected/bd4cbe7f-35b4-45df-92d7-778847de0c25-kube-api-access-87ft8\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.657191 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.657122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd4cbe7f-35b4-45df-92d7-778847de0c25-istiod-ca-cert\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.657191 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.657188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-data\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.657627 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.657218    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-podinfo\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.657627 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.657246    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-workload-certs\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.657627 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.657515    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-workload-socket\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.657806 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.657695    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-data\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.657806 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.657738    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-credential-socket\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.657806 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.657763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-workload-certs\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.657978 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.657959    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd4cbe7f-35b4-45df-92d7-778847de0c25-istiod-ca-cert\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.659482 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.659459    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-envoy\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.659872 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.659850    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-podinfo\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.666951 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.666924    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-token\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.667041 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.667014    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87ft8\" (UniqueName: \"kubernetes.io/projected/bd4cbe7f-35b4-45df-92d7-778847de0c25-kube-api-access-87ft8\") pod \"e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") " pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:55.782998 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:55.782920    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:56.071946 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:56.071894    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"]
Sep 07 13:18:56.072295 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:18:56.072268    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4cbe7f_35b4_45df_92d7_778847de0c25.slice/crio-819ebc3333550500b3c4e782dd191e1bbb833e169c0fdadbd3b9b55686faf9d2 WatchSource:0}: Error finding container 819ebc3333550500b3c4e782dd191e1bbb833e169c0fdadbd3b9b55686faf9d2: Status 404 returned error can't find the container with id 819ebc3333550500b3c4e782dd191e1bbb833e169c0fdadbd3b9b55686faf9d2
Sep 07 13:18:56.074997 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:56.074957    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:18:56.075081 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:56.075053    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:18:56.075133 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:56.075101    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:18:56.788824 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:56.788790    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck" event={"ID":"bd4cbe7f-35b4-45df-92d7-778847de0c25","Type":"ContainerStarted","Data":"ac251a90c03a7f93e7025ab8e50c1341385294250fc75e07ae9721733db6f504"}
Sep 07 13:18:56.789234 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:56.788856    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck" event={"ID":"bd4cbe7f-35b4-45df-92d7-778847de0c25","Type":"ContainerStarted","Data":"819ebc3333550500b3c4e782dd191e1bbb833e169c0fdadbd3b9b55686faf9d2"}
Sep 07 13:18:56.818628 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:56.818565    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck" podStartSLOduration=1.818550366 podStartE2EDuration="1.818550366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:18:56.816291842 +0000 UTC m=+1553.847158153" watchObservedRunningTime="2026-09-07 13:18:56.818550366 +0000 UTC m=+1553.849416679"
Sep 07 13:18:57.783932 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:57.783892    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:57.789664 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:57.789634    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:57.794603 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:57.794577    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:18:57.795468 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:18:57.795450    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:19:00.960585 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:00.960551    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"]
Sep 07 13:19:00.961021 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:00.961008    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3bc808-8d70-4bbb-9e1f-88a278834552" containerName="istio-proxy"
Sep 07 13:19:00.961067 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:00.961023    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3bc808-8d70-4bbb-9e1f-88a278834552" containerName="istio-proxy"
Sep 07 13:19:00.961119 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:00.961110    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3bc808-8d70-4bbb-9e1f-88a278834552" containerName="istio-proxy"
Sep 07 13:19:00.964611 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:00.964567    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"
Sep 07 13:19:00.967873 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:00.967853    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-688b44\""
Sep 07 13:19:00.978101 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:00.978082    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"]
Sep 07 13:19:01.010772 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.010741    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj\" (UID: \"fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"
Sep 07 13:19:01.010926 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.010871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzjxr\" (UniqueName: \"kubernetes.io/projected/fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb-kube-api-access-zzjxr\") pod \"payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj\" (UID: \"fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"
Sep 07 13:19:01.111827 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.111785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzjxr\" (UniqueName: \"kubernetes.io/projected/fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb-kube-api-access-zzjxr\") pod \"payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj\" (UID: \"fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"
Sep 07 13:19:01.112008 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.111906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj\" (UID: \"fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"
Sep 07 13:19:01.112628 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.112610    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj\" (UID: \"fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"
Sep 07 13:19:01.121230 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.121203    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzjxr\" (UniqueName: \"kubernetes.io/projected/fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb-kube-api-access-zzjxr\") pod \"payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj\" (UID: \"fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"
Sep 07 13:19:01.183708 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.183676    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc4b05-29813115-dlbjd"]
Sep 07 13:19:01.187252 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.187219    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bc4b05-29813115-dlbjd"]
Sep 07 13:19:01.223312 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.223227    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"]
Sep 07 13:19:01.223535 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.223463    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5" podUID="4aeafa44-ad23-41a6-8ce8-6d42045202d5" containerName="payload-processing" containerID="cri-o://17bca8a9ec70af7f57ddbb6247952583196ff1c3fd193b2026a35df862cfe3bc" gracePeriod=30
Sep 07 13:19:01.234928 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.234875    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"]
Sep 07 13:19:01.235153 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.235116    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g" podUID="fcc77009-2064-4063-8512-b913d71f3d4b" containerName="payload-pre-processing" containerID="cri-o://34dcc8a1cdacdde2ac039b7a09bde4612c28ad0e54b6151a6fe08a924cfc5ee3" gracePeriod=30
Sep 07 13:19:01.276255 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.276220    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"
Sep 07 13:19:01.479043 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.478982    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"
Sep 07 13:19:01.490705 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.490673    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3043b73-1ff8-4426-90d4-d140dab602dc" path="/var/lib/kubelet/pods/b3043b73-1ff8-4426-90d4-d140dab602dc/volumes"
Sep 07 13:19:01.516539 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.516505    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aeafa44-ad23-41a6-8ce8-6d42045202d5-plugins-config-volume\") pod \"4aeafa44-ad23-41a6-8ce8-6d42045202d5\" (UID: \"4aeafa44-ad23-41a6-8ce8-6d42045202d5\") "
Sep 07 13:19:01.516877 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.516857    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts22g\" (UniqueName: \"kubernetes.io/projected/4aeafa44-ad23-41a6-8ce8-6d42045202d5-kube-api-access-ts22g\") pod \"4aeafa44-ad23-41a6-8ce8-6d42045202d5\" (UID: \"4aeafa44-ad23-41a6-8ce8-6d42045202d5\") "
Sep 07 13:19:01.517803 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.517575    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aeafa44-ad23-41a6-8ce8-6d42045202d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4aeafa44-ad23-41a6-8ce8-6d42045202d5" (UID: "4aeafa44-ad23-41a6-8ce8-6d42045202d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:19:01.519913 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.519886    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aeafa44-ad23-41a6-8ce8-6d42045202d5-kube-api-access-ts22g" (OuterVolumeSpecName: "kube-api-access-ts22g") pod "4aeafa44-ad23-41a6-8ce8-6d42045202d5" (UID: "4aeafa44-ad23-41a6-8ce8-6d42045202d5"). InnerVolumeSpecName "kube-api-access-ts22g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:19:01.531827 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.531802    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"
Sep 07 13:19:01.565901 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.565368    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"]
Sep 07 13:19:01.618706 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.618664    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-869xx\" (UniqueName: \"kubernetes.io/projected/fcc77009-2064-4063-8512-b913d71f3d4b-kube-api-access-869xx\") pod \"fcc77009-2064-4063-8512-b913d71f3d4b\" (UID: \"fcc77009-2064-4063-8512-b913d71f3d4b\") "
Sep 07 13:19:01.618844 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.618723    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc77009-2064-4063-8512-b913d71f3d4b-plugins-config-volume\") pod \"fcc77009-2064-4063-8512-b913d71f3d4b\" (UID: \"fcc77009-2064-4063-8512-b913d71f3d4b\") "
Sep 07 13:19:01.619070 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.619042    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc77009-2064-4063-8512-b913d71f3d4b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fcc77009-2064-4063-8512-b913d71f3d4b" (UID: "fcc77009-2064-4063-8512-b913d71f3d4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:19:01.619192 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.619071    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aeafa44-ad23-41a6-8ce8-6d42045202d5-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:19:01.619192 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.619089    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts22g\" (UniqueName: \"kubernetes.io/projected/4aeafa44-ad23-41a6-8ce8-6d42045202d5-kube-api-access-ts22g\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:19:01.620827 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.620805    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc77009-2064-4063-8512-b913d71f3d4b-kube-api-access-869xx" (OuterVolumeSpecName: "kube-api-access-869xx") pod "fcc77009-2064-4063-8512-b913d71f3d4b" (UID: "fcc77009-2064-4063-8512-b913d71f3d4b"). InnerVolumeSpecName "kube-api-access-869xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:19:01.720075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.720040    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-869xx\" (UniqueName: \"kubernetes.io/projected/fcc77009-2064-4063-8512-b913d71f3d4b-kube-api-access-869xx\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:19:01.720075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.720072    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcc77009-2064-4063-8512-b913d71f3d4b-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:19:01.819993 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.819953    2565 generic.go:356] "Generic (PLEG): container finished" podID="fcc77009-2064-4063-8512-b913d71f3d4b" containerID="34dcc8a1cdacdde2ac039b7a09bde4612c28ad0e54b6151a6fe08a924cfc5ee3" exitCode=0
Sep 07 13:19:01.820189 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.820054    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g" event={"ID":"fcc77009-2064-4063-8512-b913d71f3d4b","Type":"ContainerDied","Data":"34dcc8a1cdacdde2ac039b7a09bde4612c28ad0e54b6151a6fe08a924cfc5ee3"}
Sep 07 13:19:01.820189 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.820062    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"
Sep 07 13:19:01.820189 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.820103    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g" event={"ID":"fcc77009-2064-4063-8512-b913d71f3d4b","Type":"ContainerDied","Data":"04fc2fe990c505ee76b5c3a5e1dc9ee10fbf6e6d273f17a568f7db7c8f479881"}
Sep 07 13:19:01.820189 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.820135    2565 scope.go:117] "RemoveContainer" containerID="34dcc8a1cdacdde2ac039b7a09bde4612c28ad0e54b6151a6fe08a924cfc5ee3"
Sep 07 13:19:01.822051 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.822027    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj" event={"ID":"fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb","Type":"ContainerStarted","Data":"385c5c51811519797ccd481d224a9a3f5adba6cdd4f52018083f3b1861d6c624"}
Sep 07 13:19:01.822178 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.822098    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj" event={"ID":"fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb","Type":"ContainerStarted","Data":"776cd10fd647e1498b8a304df17a2c5352af971eefd6eb6eca6fcf8b15a5f5bf"}
Sep 07 13:19:01.822178 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.822171    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"
Sep 07 13:19:01.823630 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.823606    2565 generic.go:356] "Generic (PLEG): container finished" podID="4aeafa44-ad23-41a6-8ce8-6d42045202d5" containerID="17bca8a9ec70af7f57ddbb6247952583196ff1c3fd193b2026a35df862cfe3bc" exitCode=0
Sep 07 13:19:01.823751 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.823663    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"
Sep 07 13:19:01.823751 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.823746    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5" event={"ID":"4aeafa44-ad23-41a6-8ce8-6d42045202d5","Type":"ContainerDied","Data":"17bca8a9ec70af7f57ddbb6247952583196ff1c3fd193b2026a35df862cfe3bc"}
Sep 07 13:19:01.823867 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.823786    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5" event={"ID":"4aeafa44-ad23-41a6-8ce8-6d42045202d5","Type":"ContainerDied","Data":"1affdcb33c22f8ab84787ecbe5e23521223fbdb80fa258ca0ed961f0ab0ffb08"}
Sep 07 13:19:01.831311 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.831102    2565 scope.go:117] "RemoveContainer" containerID="34dcc8a1cdacdde2ac039b7a09bde4612c28ad0e54b6151a6fe08a924cfc5ee3"
Sep 07 13:19:01.831461 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:19:01.831439    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34dcc8a1cdacdde2ac039b7a09bde4612c28ad0e54b6151a6fe08a924cfc5ee3\": container with ID starting with 34dcc8a1cdacdde2ac039b7a09bde4612c28ad0e54b6151a6fe08a924cfc5ee3 not found: ID does not exist" containerID="34dcc8a1cdacdde2ac039b7a09bde4612c28ad0e54b6151a6fe08a924cfc5ee3"
Sep 07 13:19:01.831534 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.831474    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34dcc8a1cdacdde2ac039b7a09bde4612c28ad0e54b6151a6fe08a924cfc5ee3"} err="failed to get container status \"34dcc8a1cdacdde2ac039b7a09bde4612c28ad0e54b6151a6fe08a924cfc5ee3\": rpc error: code = NotFound desc = could not find container \"34dcc8a1cdacdde2ac039b7a09bde4612c28ad0e54b6151a6fe08a924cfc5ee3\": container with ID starting with 34dcc8a1cdacdde2ac039b7a09bde4612c28ad0e54b6151a6fe08a924cfc5ee3 not found: ID does not exist"
Sep 07 13:19:01.831534 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.831498    2565 scope.go:117] "RemoveContainer" containerID="17bca8a9ec70af7f57ddbb6247952583196ff1c3fd193b2026a35df862cfe3bc"
Sep 07 13:19:01.840361 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.840309    2565 scope.go:117] "RemoveContainer" containerID="17bca8a9ec70af7f57ddbb6247952583196ff1c3fd193b2026a35df862cfe3bc"
Sep 07 13:19:01.840653 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:19:01.840634    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17bca8a9ec70af7f57ddbb6247952583196ff1c3fd193b2026a35df862cfe3bc\": container with ID starting with 17bca8a9ec70af7f57ddbb6247952583196ff1c3fd193b2026a35df862cfe3bc not found: ID does not exist" containerID="17bca8a9ec70af7f57ddbb6247952583196ff1c3fd193b2026a35df862cfe3bc"
Sep 07 13:19:01.840701 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.840660    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17bca8a9ec70af7f57ddbb6247952583196ff1c3fd193b2026a35df862cfe3bc"} err="failed to get container status \"17bca8a9ec70af7f57ddbb6247952583196ff1c3fd193b2026a35df862cfe3bc\": rpc error: code = NotFound desc = could not find container \"17bca8a9ec70af7f57ddbb6247952583196ff1c3fd193b2026a35df862cfe3bc\": container with ID starting with 17bca8a9ec70af7f57ddbb6247952583196ff1c3fd193b2026a35df862cfe3bc not found: ID does not exist"
Sep 07 13:19:01.844647 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.844604    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj" podStartSLOduration=1.844591695 podStartE2EDuration="1.844591695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:19:01.842620956 +0000 UTC m=+1558.873487268" watchObservedRunningTime="2026-09-07 13:19:01.844591695 +0000 UTC m=+1558.875458004"
Sep 07 13:19:01.862432 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.862409    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"]
Sep 07 13:19:01.870464 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.870442    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-bc4b05-55d4796dc7-85zr5"]
Sep 07 13:19:01.882654 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.882625    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"]
Sep 07 13:19:01.887677 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:01.887654    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-bc4b05-84f84cb87d-tg42g"]
Sep 07 13:19:03.490990 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:03.490955    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aeafa44-ad23-41a6-8ce8-6d42045202d5" path="/var/lib/kubelet/pods/4aeafa44-ad23-41a6-8ce8-6d42045202d5/volumes"
Sep 07 13:19:03.491399 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:03.491350    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcc77009-2064-4063-8512-b913d71f3d4b" path="/var/lib/kubelet/pods/fcc77009-2064-4063-8512-b913d71f3d4b/volumes"
Sep 07 13:19:08.899924 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:08.899886    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"]
Sep 07 13:19:08.900314 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:08.900080    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd" podUID="45828b52-0938-4d26-942c-28fe56c6c212" containerName="istio-proxy" containerID="cri-o://745455cac6a8b91968c306718ac040ae6e30cc88b5b7ffd0b3b64e18e44d7dda" gracePeriod=30
Sep 07 13:19:09.048760 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.048724    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:19:09.373630 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.373596    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg"]
Sep 07 13:19:09.374068 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.374055    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc77009-2064-4063-8512-b913d71f3d4b" containerName="payload-pre-processing"
Sep 07 13:19:09.374114 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.374070    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc77009-2064-4063-8512-b913d71f3d4b" containerName="payload-pre-processing"
Sep 07 13:19:09.374114 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.374100    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aeafa44-ad23-41a6-8ce8-6d42045202d5" containerName="payload-processing"
Sep 07 13:19:09.374114 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.374105    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aeafa44-ad23-41a6-8ce8-6d42045202d5" containerName="payload-processing"
Sep 07 13:19:09.374213 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.374170    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aeafa44-ad23-41a6-8ce8-6d42045202d5" containerName="payload-processing"
Sep 07 13:19:09.374213 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.374178    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcc77009-2064-4063-8512-b913d71f3d4b" containerName="payload-pre-processing"
Sep 07 13:19:09.378993 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.378976    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg"
Sep 07 13:19:09.381872 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.381842    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flngs\""
Sep 07 13:19:09.382316 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.382293    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 13:19:09.394641 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.394615    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg"]
Sep 07 13:19:09.502609 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.502575    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b47ceb-fb4f-4d08-9532-e7ed6c14e87b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg\" (UID: \"68b47ceb-fb4f-4d08-9532-e7ed6c14e87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg"
Sep 07 13:19:09.603601 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.603552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b47ceb-fb4f-4d08-9532-e7ed6c14e87b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg\" (UID: \"68b47ceb-fb4f-4d08-9532-e7ed6c14e87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg"
Sep 07 13:19:09.604213 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.604192    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b47ceb-fb4f-4d08-9532-e7ed6c14e87b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg\" (UID: \"68b47ceb-fb4f-4d08-9532-e7ed6c14e87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg"
Sep 07 13:19:09.689188 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.689106    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg"
Sep 07 13:19:09.965480 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:09.965454    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg"]
Sep 07 13:19:09.967826 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:19:09.967800    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b47ceb_fb4f_4d08_9532_e7ed6c14e87b.slice/crio-508b7f7a95ed5dfbaaf66873f51585a018ea6190b5d51c0fd159475fc86a1d99 WatchSource:0}: Error finding container 508b7f7a95ed5dfbaaf66873f51585a018ea6190b5d51c0fd159475fc86a1d99: Status 404 returned error can't find the container with id 508b7f7a95ed5dfbaaf66873f51585a018ea6190b5d51c0fd159475fc86a1d99
Sep 07 13:19:10.876895 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:10.876833    2565 generic.go:356] "Generic (PLEG): container finished" podID="68b47ceb-fb4f-4d08-9532-e7ed6c14e87b" containerID="3fc87abb0b6fd3fb452b96db21e3a5861d8f7f519043c86272af7c602cf90d2e" exitCode=0
Sep 07 13:19:10.877082 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:10.876956    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg" event={"ID":"68b47ceb-fb4f-4d08-9532-e7ed6c14e87b","Type":"ContainerDied","Data":"3fc87abb0b6fd3fb452b96db21e3a5861d8f7f519043c86272af7c602cf90d2e"}
Sep 07 13:19:10.877082 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:10.877027    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg" event={"ID":"68b47ceb-fb4f-4d08-9532-e7ed6c14e87b","Type":"ContainerStarted","Data":"508b7f7a95ed5dfbaaf66873f51585a018ea6190b5d51c0fd159475fc86a1d99"}
Sep 07 13:19:12.067082 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:12.067056    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg"
Sep 07 13:19:12.232007 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:12.231913    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b47ceb-fb4f-4d08-9532-e7ed6c14e87b-maas-api-service-ca\") pod \"68b47ceb-fb4f-4d08-9532-e7ed6c14e87b\" (UID: \"68b47ceb-fb4f-4d08-9532-e7ed6c14e87b\") "
Sep 07 13:19:12.232226 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:12.232188    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b47ceb-fb4f-4d08-9532-e7ed6c14e87b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "68b47ceb-fb4f-4d08-9532-e7ed6c14e87b" (UID: "68b47ceb-fb4f-4d08-9532-e7ed6c14e87b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:19:12.332937 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:12.332894    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b47ceb-fb4f-4d08-9532-e7ed6c14e87b-maas-api-service-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:19:12.334771 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:12.334742    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 13:19:12.334900 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:12.334799    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd" podUID="45828b52-0938-4d26-942c-28fe56c6c212" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 13:19:12.834835 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:12.834805    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"
Sep 07 13:19:12.899983 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:12.897464    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg" event={"ID":"68b47ceb-fb4f-4d08-9532-e7ed6c14e87b","Type":"ContainerDied","Data":"508b7f7a95ed5dfbaaf66873f51585a018ea6190b5d51c0fd159475fc86a1d99"}
Sep 07 13:19:12.899983 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:12.897531    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="508b7f7a95ed5dfbaaf66873f51585a018ea6190b5d51c0fd159475fc86a1d99"
Sep 07 13:19:12.899983 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:12.897603    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg"
Sep 07 13:19:14.169410 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.169348    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:19:14.250371 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.250310    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-credential-socket\") pod \"45828b52-0938-4d26-942c-28fe56c6c212\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") "
Sep 07 13:19:14.250569 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.250385    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gqjw\" (UniqueName: \"kubernetes.io/projected/45828b52-0938-4d26-942c-28fe56c6c212-kube-api-access-7gqjw\") pod \"45828b52-0938-4d26-942c-28fe56c6c212\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") "
Sep 07 13:19:14.250569 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.250416    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-istio-data\") pod \"45828b52-0938-4d26-942c-28fe56c6c212\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") "
Sep 07 13:19:14.250702 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.250603    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45828b52-0938-4d26-942c-28fe56c6c212" (UID: "45828b52-0938-4d26-942c-28fe56c6c212"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:19:14.250702 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.250622    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-istio-data" (OuterVolumeSpecName: "istio-data") pod "45828b52-0938-4d26-942c-28fe56c6c212" (UID: "45828b52-0938-4d26-942c-28fe56c6c212"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:19:14.250702 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.250691    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-istio-envoy\") pod \"45828b52-0938-4d26-942c-28fe56c6c212\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") "
Sep 07 13:19:14.250847 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.250761    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-workload-socket\") pod \"45828b52-0938-4d26-942c-28fe56c6c212\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") "
Sep 07 13:19:14.250847 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.250817    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45828b52-0938-4d26-942c-28fe56c6c212-istio-token\") pod \"45828b52-0938-4d26-942c-28fe56c6c212\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") "
Sep 07 13:19:14.250915 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.250856    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45828b52-0938-4d26-942c-28fe56c6c212-istio-podinfo\") pod \"45828b52-0938-4d26-942c-28fe56c6c212\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") "
Sep 07 13:19:14.250915 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.250891    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-workload-certs\") pod \"45828b52-0938-4d26-942c-28fe56c6c212\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") "
Sep 07 13:19:14.250991 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.250938    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45828b52-0938-4d26-942c-28fe56c6c212-istiod-ca-cert\") pod \"45828b52-0938-4d26-942c-28fe56c6c212\" (UID: \"45828b52-0938-4d26-942c-28fe56c6c212\") "
Sep 07 13:19:14.251060 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.251036    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45828b52-0938-4d26-942c-28fe56c6c212" (UID: "45828b52-0938-4d26-942c-28fe56c6c212"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:19:14.251464 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.251443    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-workload-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:19:14.251738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.251469    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-credential-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:19:14.251738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.251484    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-istio-data\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:19:14.251738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.251572    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45828b52-0938-4d26-942c-28fe56c6c212" (UID: "45828b52-0938-4d26-942c-28fe56c6c212"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:19:14.251920 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.251802    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45828b52-0938-4d26-942c-28fe56c6c212-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45828b52-0938-4d26-942c-28fe56c6c212" (UID: "45828b52-0938-4d26-942c-28fe56c6c212"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:19:14.253120 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.253096    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45828b52-0938-4d26-942c-28fe56c6c212" (UID: "45828b52-0938-4d26-942c-28fe56c6c212"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:19:14.253120 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.253108    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45828b52-0938-4d26-942c-28fe56c6c212-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45828b52-0938-4d26-942c-28fe56c6c212" (UID: "45828b52-0938-4d26-942c-28fe56c6c212"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 13:19:14.253262 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.253114    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45828b52-0938-4d26-942c-28fe56c6c212-kube-api-access-7gqjw" (OuterVolumeSpecName: "kube-api-access-7gqjw") pod "45828b52-0938-4d26-942c-28fe56c6c212" (UID: "45828b52-0938-4d26-942c-28fe56c6c212"). InnerVolumeSpecName "kube-api-access-7gqjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:19:14.254478 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.254455    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45828b52-0938-4d26-942c-28fe56c6c212-istio-token" (OuterVolumeSpecName: "istio-token") pod "45828b52-0938-4d26-942c-28fe56c6c212" (UID: "45828b52-0938-4d26-942c-28fe56c6c212"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:19:14.352910 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.352874    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45828b52-0938-4d26-942c-28fe56c6c212-istio-token\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:19:14.352910 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.352905    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45828b52-0938-4d26-942c-28fe56c6c212-istio-podinfo\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:19:14.352910 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.352914    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-workload-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:19:14.352910 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.352922    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45828b52-0938-4d26-942c-28fe56c6c212-istiod-ca-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:19:14.353186 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.352931    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gqjw\" (UniqueName: \"kubernetes.io/projected/45828b52-0938-4d26-942c-28fe56c6c212-kube-api-access-7gqjw\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:19:14.353186 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.352939    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45828b52-0938-4d26-942c-28fe56c6c212-istio-envoy\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:19:14.916633 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.916586    2565 generic.go:356] "Generic (PLEG): container finished" podID="45828b52-0938-4d26-942c-28fe56c6c212" containerID="745455cac6a8b91968c306718ac040ae6e30cc88b5b7ffd0b3b64e18e44d7dda" exitCode=0
Sep 07 13:19:14.916633 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.916638    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"
Sep 07 13:19:14.916849 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.916642    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd" event={"ID":"45828b52-0938-4d26-942c-28fe56c6c212","Type":"ContainerDied","Data":"745455cac6a8b91968c306718ac040ae6e30cc88b5b7ffd0b3b64e18e44d7dda"}
Sep 07 13:19:14.916849 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.916700    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd" event={"ID":"45828b52-0938-4d26-942c-28fe56c6c212","Type":"ContainerDied","Data":"fcb89e96dc8b8fd57061147fb018f75962a7ae4e5eb92bafdd15732ae9d36359"}
Sep 07 13:19:14.916849 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.916731    2565 scope.go:117] "RemoveContainer" containerID="745455cac6a8b91968c306718ac040ae6e30cc88b5b7ffd0b3b64e18e44d7dda"
Sep 07 13:19:14.926676 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.926660    2565 scope.go:117] "RemoveContainer" containerID="0fd34a4d783cd2240ca4cc65568dc9a504661920cd48d12148af90fdd069531f"
Sep 07 13:19:14.935508 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.935490    2565 scope.go:117] "RemoveContainer" containerID="745455cac6a8b91968c306718ac040ae6e30cc88b5b7ffd0b3b64e18e44d7dda"
Sep 07 13:19:14.935791 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:19:14.935765    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"745455cac6a8b91968c306718ac040ae6e30cc88b5b7ffd0b3b64e18e44d7dda\": container with ID starting with 745455cac6a8b91968c306718ac040ae6e30cc88b5b7ffd0b3b64e18e44d7dda not found: ID does not exist" containerID="745455cac6a8b91968c306718ac040ae6e30cc88b5b7ffd0b3b64e18e44d7dda"
Sep 07 13:19:14.935891 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.935797    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745455cac6a8b91968c306718ac040ae6e30cc88b5b7ffd0b3b64e18e44d7dda"} err="failed to get container status \"745455cac6a8b91968c306718ac040ae6e30cc88b5b7ffd0b3b64e18e44d7dda\": rpc error: code = NotFound desc = could not find container \"745455cac6a8b91968c306718ac040ae6e30cc88b5b7ffd0b3b64e18e44d7dda\": container with ID starting with 745455cac6a8b91968c306718ac040ae6e30cc88b5b7ffd0b3b64e18e44d7dda not found: ID does not exist"
Sep 07 13:19:14.935891 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.935815    2565 scope.go:117] "RemoveContainer" containerID="0fd34a4d783cd2240ca4cc65568dc9a504661920cd48d12148af90fdd069531f"
Sep 07 13:19:14.936048 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:19:14.936027    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd34a4d783cd2240ca4cc65568dc9a504661920cd48d12148af90fdd069531f\": container with ID starting with 0fd34a4d783cd2240ca4cc65568dc9a504661920cd48d12148af90fdd069531f not found: ID does not exist" containerID="0fd34a4d783cd2240ca4cc65568dc9a504661920cd48d12148af90fdd069531f"
Sep 07 13:19:14.936087 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.936056    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd34a4d783cd2240ca4cc65568dc9a504661920cd48d12148af90fdd069531f"} err="failed to get container status \"0fd34a4d783cd2240ca4cc65568dc9a504661920cd48d12148af90fdd069531f\": rpc error: code = NotFound desc = could not find container \"0fd34a4d783cd2240ca4cc65568dc9a504661920cd48d12148af90fdd069531f\": container with ID starting with 0fd34a4d783cd2240ca4cc65568dc9a504661920cd48d12148af90fdd069531f not found: ID does not exist"
Sep 07 13:19:14.941535 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.941512    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"]
Sep 07 13:19:14.958050 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:14.957472    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bc4b05-openshift-default-7c48b9d7b6-x5kvd"]
Sep 07 13:19:15.494686 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:15.494643    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45828b52-0938-4d26-942c-28fe56c6c212" path="/var/lib/kubelet/pods/45828b52-0938-4d26-942c-28fe56c6c212/volumes"
Sep 07 13:19:19.907707 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:19.907673    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"]
Sep 07 13:19:19.908209 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:19.907981    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj" podUID="32e78db2-72cb-498e-8375-50e4b30de76e" containerName="payload-pre-processing" containerID="cri-o://1e7eb689326c6902816625e69db105a2b8c2654f8d97d9e2413c73e14f865185" gracePeriod=30
Sep 07 13:19:19.958358 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:19.958294    2565 generic.go:356] "Generic (PLEG): container finished" podID="bd4cbe7f-35b4-45df-92d7-778847de0c25" containerID="ac251a90c03a7f93e7025ab8e50c1341385294250fc75e07ae9721733db6f504" exitCode=0
Sep 07 13:19:19.958538 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:19.958378    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck" event={"ID":"bd4cbe7f-35b4-45df-92d7-778847de0c25","Type":"ContainerDied","Data":"ac251a90c03a7f93e7025ab8e50c1341385294250fc75e07ae9721733db6f504"}
Sep 07 13:19:19.958874 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:19.958856    2565 scope.go:117] "RemoveContainer" containerID="ac251a90c03a7f93e7025ab8e50c1341385294250fc75e07ae9721733db6f504"
Sep 07 13:19:19.960171 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:19.960136    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:19:19.960294 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:19.960211    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:19:19.960294 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:19.960242    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:19:20.110063 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.109169    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg"]
Sep 07 13:19:20.112411 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.112379    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-45d721-qlbbg"]
Sep 07 13:19:20.169674 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.169600    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"
Sep 07 13:19:20.314697 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.314666    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4w4p\" (UniqueName: \"kubernetes.io/projected/32e78db2-72cb-498e-8375-50e4b30de76e-kube-api-access-w4w4p\") pod \"32e78db2-72cb-498e-8375-50e4b30de76e\" (UID: \"32e78db2-72cb-498e-8375-50e4b30de76e\") "
Sep 07 13:19:20.314883 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.314715    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32e78db2-72cb-498e-8375-50e4b30de76e-plugins-config-volume\") pod \"32e78db2-72cb-498e-8375-50e4b30de76e\" (UID: \"32e78db2-72cb-498e-8375-50e4b30de76e\") "
Sep 07 13:19:20.315096 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.315072    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e78db2-72cb-498e-8375-50e4b30de76e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "32e78db2-72cb-498e-8375-50e4b30de76e" (UID: "32e78db2-72cb-498e-8375-50e4b30de76e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:19:20.317289 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.317262    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e78db2-72cb-498e-8375-50e4b30de76e-kube-api-access-w4w4p" (OuterVolumeSpecName: "kube-api-access-w4w4p") pod "32e78db2-72cb-498e-8375-50e4b30de76e" (UID: "32e78db2-72cb-498e-8375-50e4b30de76e"). InnerVolumeSpecName "kube-api-access-w4w4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:19:20.416368 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.416301    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4w4p\" (UniqueName: \"kubernetes.io/projected/32e78db2-72cb-498e-8375-50e4b30de76e-kube-api-access-w4w4p\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:19:20.416368 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.416362    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32e78db2-72cb-498e-8375-50e4b30de76e-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:19:20.966170 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.966133    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck" event={"ID":"bd4cbe7f-35b4-45df-92d7-778847de0c25","Type":"ContainerStarted","Data":"1883dddaf824f900aca364dd8e6ad97a34b505c962b63f8fa97b5269293b512a"}
Sep 07 13:19:20.967846 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.967813    2565 generic.go:356] "Generic (PLEG): container finished" podID="32e78db2-72cb-498e-8375-50e4b30de76e" containerID="1e7eb689326c6902816625e69db105a2b8c2654f8d97d9e2413c73e14f865185" exitCode=0
Sep 07 13:19:20.967999 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.967849    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"
Sep 07 13:19:20.967999 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.967895    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj" event={"ID":"32e78db2-72cb-498e-8375-50e4b30de76e","Type":"ContainerDied","Data":"1e7eb689326c6902816625e69db105a2b8c2654f8d97d9e2413c73e14f865185"}
Sep 07 13:19:20.967999 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.967952    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj" event={"ID":"32e78db2-72cb-498e-8375-50e4b30de76e","Type":"ContainerDied","Data":"aee9cee2c741cbd412029a6a0beb27f87e0ccce7dd86379894fa0b9970180b82"}
Sep 07 13:19:20.967999 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.967981    2565 scope.go:117] "RemoveContainer" containerID="1e7eb689326c6902816625e69db105a2b8c2654f8d97d9e2413c73e14f865185"
Sep 07 13:19:20.977664 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.977647    2565 scope.go:117] "RemoveContainer" containerID="1e7eb689326c6902816625e69db105a2b8c2654f8d97d9e2413c73e14f865185"
Sep 07 13:19:20.977976 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:19:20.977957    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e7eb689326c6902816625e69db105a2b8c2654f8d97d9e2413c73e14f865185\": container with ID starting with 1e7eb689326c6902816625e69db105a2b8c2654f8d97d9e2413c73e14f865185 not found: ID does not exist" containerID="1e7eb689326c6902816625e69db105a2b8c2654f8d97d9e2413c73e14f865185"
Sep 07 13:19:20.978044 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:20.977986    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e7eb689326c6902816625e69db105a2b8c2654f8d97d9e2413c73e14f865185"} err="failed to get container status \"1e7eb689326c6902816625e69db105a2b8c2654f8d97d9e2413c73e14f865185\": rpc error: code = NotFound desc = could not find container \"1e7eb689326c6902816625e69db105a2b8c2654f8d97d9e2413c73e14f865185\": container with ID starting with 1e7eb689326c6902816625e69db105a2b8c2654f8d97d9e2413c73e14f865185 not found: ID does not exist"
Sep 07 13:19:21.009719 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:21.009669    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"]
Sep 07 13:19:21.012507 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:21.012476    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-45d721-cd9f5b794-mvmtj"]
Sep 07 13:19:21.493186 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:21.493142    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32e78db2-72cb-498e-8375-50e4b30de76e" path="/var/lib/kubelet/pods/32e78db2-72cb-498e-8375-50e4b30de76e/volumes"
Sep 07 13:19:21.493704 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:21.493678    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68b47ceb-fb4f-4d08-9532-e7ed6c14e87b" path="/var/lib/kubelet/pods/68b47ceb-fb4f-4d08-9532-e7ed6c14e87b/volumes"
Sep 07 13:19:21.783135 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:21.783036    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:19:21.783387 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:21.783359    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 07 13:19:21.783489 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:21.783432    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck" podUID="bd4cbe7f-35b4-45df-92d7-778847de0c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 07 13:19:22.783675 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:22.783629    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 07 13:19:22.784176 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:22.783706    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck" podUID="bd4cbe7f-35b4-45df-92d7-778847de0c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 07 13:19:23.784282 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:23.784230    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 07 13:19:23.784820 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:23.784307    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck" podUID="bd4cbe7f-35b4-45df-92d7-778847de0c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 07 13:19:24.784209 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:24.784168    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 07 13:19:24.784589 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:24.784555    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck" podUID="bd4cbe7f-35b4-45df-92d7-778847de0c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 07 13:19:25.512852 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:25.512816    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:19:25.784034 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:25.783928    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 07 13:19:25.784034 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:25.784021    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck" podUID="bd4cbe7f-35b4-45df-92d7-778847de0c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 07 13:19:26.787539 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:26.787507    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:19:26.788037 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:26.787989    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:19:26.788783 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:26.788764    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:19:38.514613 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.514575    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"]
Sep 07 13:19:38.515042 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.515013    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68b47ceb-fb4f-4d08-9532-e7ed6c14e87b" containerName="revoke-keys"
Sep 07 13:19:38.515042 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.515031    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="68b47ceb-fb4f-4d08-9532-e7ed6c14e87b" containerName="revoke-keys"
Sep 07 13:19:38.515042 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.515045    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45828b52-0938-4d26-942c-28fe56c6c212" containerName="istio-proxy"
Sep 07 13:19:38.515289 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.515053    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="45828b52-0938-4d26-942c-28fe56c6c212" containerName="istio-proxy"
Sep 07 13:19:38.515289 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.515062    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45828b52-0938-4d26-942c-28fe56c6c212" containerName="istio-proxy"
Sep 07 13:19:38.515289 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.515069    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="45828b52-0938-4d26-942c-28fe56c6c212" containerName="istio-proxy"
Sep 07 13:19:38.515289 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.515081    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32e78db2-72cb-498e-8375-50e4b30de76e" containerName="payload-pre-processing"
Sep 07 13:19:38.515289 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.515088    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="32e78db2-72cb-498e-8375-50e4b30de76e" containerName="payload-pre-processing"
Sep 07 13:19:38.515289 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.515219    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="45828b52-0938-4d26-942c-28fe56c6c212" containerName="istio-proxy"
Sep 07 13:19:38.515289 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.515233    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="32e78db2-72cb-498e-8375-50e4b30de76e" containerName="payload-pre-processing"
Sep 07 13:19:38.515289 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.515244    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="68b47ceb-fb4f-4d08-9532-e7ed6c14e87b" containerName="revoke-keys"
Sep 07 13:19:38.518675 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.518652    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.521230 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.521209    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ec10ff\"/\"kube-root-ca.crt\""
Sep 07 13:19:38.522045 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.522025    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ec10ff\"/\"openshift-service-ca.crt\""
Sep 07 13:19:38.522149 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.522101    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ec10ff\"/\"default-dockercfg-q7nd8\""
Sep 07 13:19:38.522213 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.522101    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ec10ff\"/\"ipp-route-ec10ff-kserve-self-signed-certs\""
Sep 07 13:19:38.529012 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.528987    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"]
Sep 07 13:19:38.593462 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.593425    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-tmp-dir\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.593462 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.593467    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-home\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.593670 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.593506    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-model-cache\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.593670 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.593537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25fs7\" (UniqueName: \"kubernetes.io/projected/ff951c43-9201-447b-af44-1c3dcfae1fd6-kube-api-access-25fs7\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.593670 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.593571    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-dshm\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.593670 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.593622    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff951c43-9201-447b-af44-1c3dcfae1fd6-tls-certs\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.695242 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.695204    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-tmp-dir\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.695475 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.695250    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-home\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.695475 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.695284    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-model-cache\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.695475 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.695344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25fs7\" (UniqueName: \"kubernetes.io/projected/ff951c43-9201-447b-af44-1c3dcfae1fd6-kube-api-access-25fs7\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.695475 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.695383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-dshm\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.695475 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.695431    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff951c43-9201-447b-af44-1c3dcfae1fd6-tls-certs\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.695759 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.695718    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-model-cache\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.695816 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.695763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-tmp-dir\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.695816 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.695793    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-home\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.697873 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.697846    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-dshm\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.698093 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.698075    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff951c43-9201-447b-af44-1c3dcfae1fd6-tls-certs\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.705089 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.705064    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25fs7\" (UniqueName: \"kubernetes.io/projected/ff951c43-9201-447b-af44-1c3dcfae1fd6-kube-api-access-25fs7\") pod \"ipp-route-ec10ff-kserve-67f6b66d75-5znx7\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") " pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:38.829349 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:38.829294    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:39.113742 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:39.113597    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"]
Sep 07 13:19:39.116316 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:19:39.116276    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff951c43_9201_447b_af44_1c3dcfae1fd6.slice/crio-3ab4280febfffd6cbf4d17985ac21a21b954a3c19360774997ba06edf2e613d7 WatchSource:0}: Error finding container 3ab4280febfffd6cbf4d17985ac21a21b954a3c19360774997ba06edf2e613d7: Status 404 returned error can't find the container with id 3ab4280febfffd6cbf4d17985ac21a21b954a3c19360774997ba06edf2e613d7
Sep 07 13:19:40.091971 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:40.091934    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7" event={"ID":"ff951c43-9201-447b-af44-1c3dcfae1fd6","Type":"ContainerStarted","Data":"5676c46f1b8461d4da6a896d9fd1600e8d6a878c2062c3bc57f73383617fe73d"}
Sep 07 13:19:40.092387 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:40.091979    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7" event={"ID":"ff951c43-9201-447b-af44-1c3dcfae1fd6","Type":"ContainerStarted","Data":"3ab4280febfffd6cbf4d17985ac21a21b954a3c19360774997ba06edf2e613d7"}
Sep 07 13:19:40.111719 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:40.111665    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7" podStartSLOduration=2.111650841 podStartE2EDuration="2.111650841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:19:40.109636716 +0000 UTC m=+1597.140503039" watchObservedRunningTime="2026-09-07 13:19:40.111650841 +0000 UTC m=+1597.142517141"
Sep 07 13:19:43.810451 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:43.810411    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:19:48.830278 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:48.830235    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:48.830703 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:48.830299    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:48.843612 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:48.843584    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:49.152026 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:49.151938    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:19:55.503020 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:19:55.502928    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:20:01.118152 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:01.118104    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw_52bc72f1-b2f2-4221-98aa-50e9af10db28/payload-processing/0.log"
Sep 07 13:20:02.030729 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.030694    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"]
Sep 07 13:20:02.031252 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.031237    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="45828b52-0938-4d26-942c-28fe56c6c212" containerName="istio-proxy"
Sep 07 13:20:02.035642 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.035622    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.038016 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.037995    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-a2a8d5-openshift-default-dockercfg-dzxb5\""
Sep 07 13:20:02.046878 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.046853    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"]
Sep 07 13:20:02.141525 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.141487    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-workload-socket\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.141525 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.141528    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-credential-socket\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.141525 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.141546    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1023b67-5901-4482-8886-8912fd062eda-istio-podinfo\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.142136 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.141561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-workload-certs\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.142136 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.141669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbgh2\" (UniqueName: \"kubernetes.io/projected/f1023b67-5901-4482-8886-8912fd062eda-kube-api-access-jbgh2\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.142136 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.141711    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1023b67-5901-4482-8886-8912fd062eda-istio-token\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.142136 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.141759    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-istio-data\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.142136 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.141788    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1023b67-5901-4482-8886-8912fd062eda-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.142136 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.141840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-istio-envoy\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.242744 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.242701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1023b67-5901-4482-8886-8912fd062eda-istio-token\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.242967 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.242775    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-istio-data\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.242967 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.242821    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1023b67-5901-4482-8886-8912fd062eda-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.242967 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.242892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-istio-envoy\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.243146 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.243033    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-workload-socket\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.243146 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.243070    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-credential-socket\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.243146 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.243097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1023b67-5901-4482-8886-8912fd062eda-istio-podinfo\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.243146 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.243122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-workload-certs\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.243395 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.243233    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbgh2\" (UniqueName: \"kubernetes.io/projected/f1023b67-5901-4482-8886-8912fd062eda-kube-api-access-jbgh2\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.243639 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.243585    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-workload-socket\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.243803 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.243781    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-credential-socket\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.243974 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.243833    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-istio-data\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.244117 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.243991    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1023b67-5901-4482-8886-8912fd062eda-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.244222 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.244013    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-workload-certs\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.246111 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.246089    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-istio-envoy\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.246576 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.246555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1023b67-5901-4482-8886-8912fd062eda-istio-podinfo\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.253835 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.253803    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1023b67-5901-4482-8886-8912fd062eda-istio-token\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.255319 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.255295    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbgh2\" (UniqueName: \"kubernetes.io/projected/f1023b67-5901-4482-8886-8912fd062eda-kube-api-access-jbgh2\") pod \"e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") " pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.347223 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.347183    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:02.627381 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.627355    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"]
Sep 07 13:20:02.633661 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.633623    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:20:02.633767 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.633708    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:20:02.633767 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:02.633750    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:20:03.214616 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:03.214580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5" event={"ID":"f1023b67-5901-4482-8886-8912fd062eda","Type":"ContainerStarted","Data":"c25f1fbddcbb5b4c36d3357f004c5a09a53fdb1a0e2e43aa230d78b0ce3aa590"}
Sep 07 13:20:03.215117 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:03.214638    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5" event={"ID":"f1023b67-5901-4482-8886-8912fd062eda","Type":"ContainerStarted","Data":"335a48325b48463297ef8422c9d29e94f1750b8923bb02789e3f29a921f86fd7"}
Sep 07 13:20:03.239280 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:03.239219    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5" podStartSLOduration=1.239202578 podStartE2EDuration="1.239202578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:20:03.236880034 +0000 UTC m=+1620.267746345" watchObservedRunningTime="2026-09-07 13:20:03.239202578 +0000 UTC m=+1620.270068888"
Sep 07 13:20:03.348143 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:03.348106    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:03.358248 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:03.358216    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:03.737568 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:03.737529    2565 scope.go:117] "RemoveContainer" containerID="043fa1f27e446f591c9a6b0f16f7cffa40c1f88d7c75798fa5cb22716b69a885"
Sep 07 13:20:04.220321 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:04.220283    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:04.221261 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:04.221242    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:12.209348 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:12.209293    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:20:23.335716 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:23.335675    2565 generic.go:356] "Generic (PLEG): container finished" podID="f1023b67-5901-4482-8886-8912fd062eda" containerID="c25f1fbddcbb5b4c36d3357f004c5a09a53fdb1a0e2e43aa230d78b0ce3aa590" exitCode=0
Sep 07 13:20:23.336160 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:23.335749    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5" event={"ID":"f1023b67-5901-4482-8886-8912fd062eda","Type":"ContainerDied","Data":"c25f1fbddcbb5b4c36d3357f004c5a09a53fdb1a0e2e43aa230d78b0ce3aa590"}
Sep 07 13:20:23.336206 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:23.336160    2565 scope.go:117] "RemoveContainer" containerID="c25f1fbddcbb5b4c36d3357f004c5a09a53fdb1a0e2e43aa230d78b0ce3aa590"
Sep 07 13:20:23.337253 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:23.337225    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:20:23.337369 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:23.337290    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:20:23.337369 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:23.337319    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 07 13:20:24.343358 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:24.343302    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5" event={"ID":"f1023b67-5901-4482-8886-8912fd062eda","Type":"ContainerStarted","Data":"e5bd713cc008b1409aac2e4d98fcf71dd3e985420b1a7c87d1541ca5f40c76a5"}
Sep 07 13:20:24.347372 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:24.347352    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:24.347594 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:24.347568    2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 07 13:20:24.347646 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:24.347614    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5" podUID="f1023b67-5901-4482-8886-8912fd062eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 07 13:20:25.347912 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:25.347878    2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 07 13:20:25.348304 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:25.347941    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5" podUID="f1023b67-5901-4482-8886-8912fd062eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 07 13:20:26.348089 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:26.348052    2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 07 13:20:26.348535 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:26.348119    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5" podUID="f1023b67-5901-4482-8886-8912fd062eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 07 13:20:27.347685 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:27.347638    2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 07 13:20:27.347879 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:27.347701    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5" podUID="f1023b67-5901-4482-8886-8912fd062eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 07 13:20:28.348345 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:28.348290    2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 07 13:20:28.348751 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:28.348392    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5" podUID="f1023b67-5901-4482-8886-8912fd062eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 07 13:20:29.348435 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:29.348386    2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 07 13:20:29.348924 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:29.348453    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5" podUID="f1023b67-5901-4482-8886-8912fd062eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 07 13:20:30.352349 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:30.352301    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:30.352815 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:30.352802    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:30.353672 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:30.353652    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:31.136020 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:31.135985    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg"]
Sep 07 13:20:31.139854 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:31.139837    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg"
Sep 07 13:20:31.142578 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:31.142555    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 13:20:31.142715 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:31.142671    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flngs\""
Sep 07 13:20:31.150087 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:31.150059    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg"]
Sep 07 13:20:31.150219 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:31.150111    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3fea46d-0b54-4f1a-abbf-4c57878347a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg\" (UID: \"b3fea46d-0b54-4f1a-abbf-4c57878347a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg"
Sep 07 13:20:31.251235 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:31.251199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3fea46d-0b54-4f1a-abbf-4c57878347a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg\" (UID: \"b3fea46d-0b54-4f1a-abbf-4c57878347a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg"
Sep 07 13:20:31.251870 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:31.251848    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3fea46d-0b54-4f1a-abbf-4c57878347a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg\" (UID: \"b3fea46d-0b54-4f1a-abbf-4c57878347a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg"
Sep 07 13:20:31.451055 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:31.450969    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg"
Sep 07 13:20:31.729084 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:31.729052    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg"]
Sep 07 13:20:31.731080 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:20:31.731039    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3fea46d_0b54_4f1a_abbf_4c57878347a4.slice/crio-6ae61c7bb3d526c1f33fce3d68635798f989eefd49a061899dd84527bbede84e WatchSource:0}: Error finding container 6ae61c7bb3d526c1f33fce3d68635798f989eefd49a061899dd84527bbede84e: Status 404 returned error can't find the container with id 6ae61c7bb3d526c1f33fce3d68635798f989eefd49a061899dd84527bbede84e
Sep 07 13:20:32.391670 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:32.391631    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg" event={"ID":"b3fea46d-0b54-4f1a-abbf-4c57878347a4","Type":"ContainerStarted","Data":"2ab3189dfb5487808aeb0b8fff8281694eaffee2a4539245a79b703d5fdd244f"}
Sep 07 13:20:32.391852 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:32.391702    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg" event={"ID":"b3fea46d-0b54-4f1a-abbf-4c57878347a4","Type":"ContainerStarted","Data":"6ae61c7bb3d526c1f33fce3d68635798f989eefd49a061899dd84527bbede84e"}
Sep 07 13:20:32.408064 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:32.407993    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg" podStartSLOduration=1.407974205 podStartE2EDuration="1.407974205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:20:32.407923717 +0000 UTC m=+1649.438790025" watchObservedRunningTime="2026-09-07 13:20:32.407974205 +0000 UTC m=+1649.438840509"
Sep 07 13:20:33.398100 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:33.398065    2565 generic.go:356] "Generic (PLEG): container finished" podID="b3fea46d-0b54-4f1a-abbf-4c57878347a4" containerID="2ab3189dfb5487808aeb0b8fff8281694eaffee2a4539245a79b703d5fdd244f" exitCode=0
Sep 07 13:20:33.398499 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:33.398153    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg" event={"ID":"b3fea46d-0b54-4f1a-abbf-4c57878347a4","Type":"ContainerDied","Data":"2ab3189dfb5487808aeb0b8fff8281694eaffee2a4539245a79b703d5fdd244f"}
Sep 07 13:20:34.539040 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:34.539012    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg"
Sep 07 13:20:34.585549 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:34.585517    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3fea46d-0b54-4f1a-abbf-4c57878347a4-maas-api-service-ca\") pod \"b3fea46d-0b54-4f1a-abbf-4c57878347a4\" (UID: \"b3fea46d-0b54-4f1a-abbf-4c57878347a4\") "
Sep 07 13:20:34.585883 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:34.585861    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3fea46d-0b54-4f1a-abbf-4c57878347a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3fea46d-0b54-4f1a-abbf-4c57878347a4" (UID: "b3fea46d-0b54-4f1a-abbf-4c57878347a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:20:34.686865 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:34.686769    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3fea46d-0b54-4f1a-abbf-4c57878347a4-maas-api-service-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:20:35.410170 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:35.410143    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg"
Sep 07 13:20:35.410378 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:35.410142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg" event={"ID":"b3fea46d-0b54-4f1a-abbf-4c57878347a4","Type":"ContainerDied","Data":"6ae61c7bb3d526c1f33fce3d68635798f989eefd49a061899dd84527bbede84e"}
Sep 07 13:20:35.410378 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:35.410292    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ae61c7bb3d526c1f33fce3d68635798f989eefd49a061899dd84527bbede84e"
Sep 07 13:20:37.477946 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:37.477909    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg"]
Sep 07 13:20:37.481276 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:37.481250    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a2a8d5-5tfpg"]
Sep 07 13:20:37.489723 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:37.489692    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3fea46d-0b54-4f1a-abbf-4c57878347a4" path="/var/lib/kubelet/pods/b3fea46d-0b54-4f1a-abbf-4c57878347a4/volumes"
Sep 07 13:20:42.220774 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:42.220740    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:20:44.870834 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:44.870801    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"]
Sep 07 13:20:44.871239 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:44.871009    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5" podUID="f1023b67-5901-4482-8886-8912fd062eda" containerName="istio-proxy" containerID="cri-o://e5bd713cc008b1409aac2e4d98fcf71dd3e985420b1a7c87d1541ca5f40c76a5" gracePeriod=30
Sep 07 13:20:45.355432 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:45.355398    2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 13:20:45.355605 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:45.355471    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5" podUID="f1023b67-5901-4482-8886-8912fd062eda" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 13:20:50.130678 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.130652    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:50.244446 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.244368    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1023b67-5901-4482-8886-8912fd062eda-istio-token\") pod \"f1023b67-5901-4482-8886-8912fd062eda\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") "
Sep 07 13:20:50.244446 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.244418    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-credential-socket\") pod \"f1023b67-5901-4482-8886-8912fd062eda\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") "
Sep 07 13:20:50.244631 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.244535    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1023b67-5901-4482-8886-8912fd062eda-istio-podinfo\") pod \"f1023b67-5901-4482-8886-8912fd062eda\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") "
Sep 07 13:20:50.244631 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.244592    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-workload-certs\") pod \"f1023b67-5901-4482-8886-8912fd062eda\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") "
Sep 07 13:20:50.244631 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.244623    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-istio-data\") pod \"f1023b67-5901-4482-8886-8912fd062eda\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") "
Sep 07 13:20:50.244797 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.244667    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-workload-socket\") pod \"f1023b67-5901-4482-8886-8912fd062eda\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") "
Sep 07 13:20:50.244797 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.244702    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f1023b67-5901-4482-8886-8912fd062eda" (UID: "f1023b67-5901-4482-8886-8912fd062eda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:20:50.244797 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.244730    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbgh2\" (UniqueName: \"kubernetes.io/projected/f1023b67-5901-4482-8886-8912fd062eda-kube-api-access-jbgh2\") pod \"f1023b67-5901-4482-8886-8912fd062eda\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") "
Sep 07 13:20:50.244797 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.244757    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-istio-envoy\") pod \"f1023b67-5901-4482-8886-8912fd062eda\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") "
Sep 07 13:20:50.245032 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.244819    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1023b67-5901-4482-8886-8912fd062eda-istiod-ca-cert\") pod \"f1023b67-5901-4482-8886-8912fd062eda\" (UID: \"f1023b67-5901-4482-8886-8912fd062eda\") "
Sep 07 13:20:50.245032 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.244883    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f1023b67-5901-4482-8886-8912fd062eda" (UID: "f1023b67-5901-4482-8886-8912fd062eda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:20:50.245032 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.244898    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-istio-data" (OuterVolumeSpecName: "istio-data") pod "f1023b67-5901-4482-8886-8912fd062eda" (UID: "f1023b67-5901-4482-8886-8912fd062eda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:20:50.245032 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.244926    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f1023b67-5901-4482-8886-8912fd062eda" (UID: "f1023b67-5901-4482-8886-8912fd062eda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:20:50.245231 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.245199    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-workload-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:20:50.245231 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.245217    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-credential-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:20:50.245352 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.245233    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-workload-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:20:50.245352 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.245247    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-istio-data\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:20:50.245577 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.245498    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1023b67-5901-4482-8886-8912fd062eda-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f1023b67-5901-4482-8886-8912fd062eda" (UID: "f1023b67-5901-4482-8886-8912fd062eda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:20:50.247086 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.247053    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1023b67-5901-4482-8886-8912fd062eda-kube-api-access-jbgh2" (OuterVolumeSpecName: "kube-api-access-jbgh2") pod "f1023b67-5901-4482-8886-8912fd062eda" (UID: "f1023b67-5901-4482-8886-8912fd062eda"). InnerVolumeSpecName "kube-api-access-jbgh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:20:50.247086 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.247076    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1023b67-5901-4482-8886-8912fd062eda-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f1023b67-5901-4482-8886-8912fd062eda" (UID: "f1023b67-5901-4482-8886-8912fd062eda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 13:20:50.247264 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.247141    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1023b67-5901-4482-8886-8912fd062eda-istio-token" (OuterVolumeSpecName: "istio-token") pod "f1023b67-5901-4482-8886-8912fd062eda" (UID: "f1023b67-5901-4482-8886-8912fd062eda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:20:50.247352 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.247304    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f1023b67-5901-4482-8886-8912fd062eda" (UID: "f1023b67-5901-4482-8886-8912fd062eda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:20:50.346853 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.346820    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbgh2\" (UniqueName: \"kubernetes.io/projected/f1023b67-5901-4482-8886-8912fd062eda-kube-api-access-jbgh2\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:20:50.346853 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.346852    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1023b67-5901-4482-8886-8912fd062eda-istio-envoy\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:20:50.347077 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.346864    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1023b67-5901-4482-8886-8912fd062eda-istiod-ca-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:20:50.347077 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.346876    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1023b67-5901-4482-8886-8912fd062eda-istio-token\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:20:50.347077 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.346887    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1023b67-5901-4482-8886-8912fd062eda-istio-podinfo\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:20:50.493262 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.493228    2565 generic.go:356] "Generic (PLEG): container finished" podID="f1023b67-5901-4482-8886-8912fd062eda" containerID="e5bd713cc008b1409aac2e4d98fcf71dd3e985420b1a7c87d1541ca5f40c76a5" exitCode=0
Sep 07 13:20:50.493445 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.493275    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"
Sep 07 13:20:50.493445 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.493317    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5" event={"ID":"f1023b67-5901-4482-8886-8912fd062eda","Type":"ContainerDied","Data":"e5bd713cc008b1409aac2e4d98fcf71dd3e985420b1a7c87d1541ca5f40c76a5"}
Sep 07 13:20:50.493445 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.493391    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5" event={"ID":"f1023b67-5901-4482-8886-8912fd062eda","Type":"ContainerDied","Data":"335a48325b48463297ef8422c9d29e94f1750b8923bb02789e3f29a921f86fd7"}
Sep 07 13:20:50.493445 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.493420    2565 scope.go:117] "RemoveContainer" containerID="e5bd713cc008b1409aac2e4d98fcf71dd3e985420b1a7c87d1541ca5f40c76a5"
Sep 07 13:20:50.503303 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.503285    2565 scope.go:117] "RemoveContainer" containerID="c25f1fbddcbb5b4c36d3357f004c5a09a53fdb1a0e2e43aa230d78b0ce3aa590"
Sep 07 13:20:50.511788 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.511773    2565 scope.go:117] "RemoveContainer" containerID="e5bd713cc008b1409aac2e4d98fcf71dd3e985420b1a7c87d1541ca5f40c76a5"
Sep 07 13:20:50.512047 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:20:50.512029    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5bd713cc008b1409aac2e4d98fcf71dd3e985420b1a7c87d1541ca5f40c76a5\": container with ID starting with e5bd713cc008b1409aac2e4d98fcf71dd3e985420b1a7c87d1541ca5f40c76a5 not found: ID does not exist" containerID="e5bd713cc008b1409aac2e4d98fcf71dd3e985420b1a7c87d1541ca5f40c76a5"
Sep 07 13:20:50.512122 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.512056    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5bd713cc008b1409aac2e4d98fcf71dd3e985420b1a7c87d1541ca5f40c76a5"} err="failed to get container status \"e5bd713cc008b1409aac2e4d98fcf71dd3e985420b1a7c87d1541ca5f40c76a5\": rpc error: code = NotFound desc = could not find container \"e5bd713cc008b1409aac2e4d98fcf71dd3e985420b1a7c87d1541ca5f40c76a5\": container with ID starting with e5bd713cc008b1409aac2e4d98fcf71dd3e985420b1a7c87d1541ca5f40c76a5 not found: ID does not exist"
Sep 07 13:20:50.512122 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.512071    2565 scope.go:117] "RemoveContainer" containerID="c25f1fbddcbb5b4c36d3357f004c5a09a53fdb1a0e2e43aa230d78b0ce3aa590"
Sep 07 13:20:50.512378 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:20:50.512308    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c25f1fbddcbb5b4c36d3357f004c5a09a53fdb1a0e2e43aa230d78b0ce3aa590\": container with ID starting with c25f1fbddcbb5b4c36d3357f004c5a09a53fdb1a0e2e43aa230d78b0ce3aa590 not found: ID does not exist" containerID="c25f1fbddcbb5b4c36d3357f004c5a09a53fdb1a0e2e43aa230d78b0ce3aa590"
Sep 07 13:20:50.512378 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.512362    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25f1fbddcbb5b4c36d3357f004c5a09a53fdb1a0e2e43aa230d78b0ce3aa590"} err="failed to get container status \"c25f1fbddcbb5b4c36d3357f004c5a09a53fdb1a0e2e43aa230d78b0ce3aa590\": rpc error: code = NotFound desc = could not find container \"c25f1fbddcbb5b4c36d3357f004c5a09a53fdb1a0e2e43aa230d78b0ce3aa590\": container with ID starting with c25f1fbddcbb5b4c36d3357f004c5a09a53fdb1a0e2e43aa230d78b0ce3aa590 not found: ID does not exist"
Sep 07 13:20:50.517801 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.517777    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"]
Sep 07 13:20:50.524707 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:50.524685    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a2a8d5-openshift-default-7479b8f8f8-cf8j5"]
Sep 07 13:20:51.489180 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:20:51.489145    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1023b67-5901-4482-8886-8912fd062eda" path="/var/lib/kubelet/pods/f1023b67-5901-4482-8886-8912fd062eda/volumes"
Sep 07 13:21:03.767716 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:03.767681    2565 scope.go:117] "RemoveContainer" containerID="7619620d123d7469ba1f4fb07ad74ecef983d769cb3380f513d8071a956e437c"
Sep 07 13:21:03.776595 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:03.776573    2565 scope.go:117] "RemoveContainer" containerID="a16e117379e4e7e62466f74e4362233ed3e6b1586a302b84ba3f5a1c44d65893"
Sep 07 13:21:04.646701 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:04.646667    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"]
Sep 07 13:21:04.647008 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:04.646949    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw" podUID="52bc72f1-b2f2-4221-98aa-50e9af10db28" containerName="payload-processing" containerID="cri-o://4137c757fad7e02bca56884d30c88af9ad6cc9ef10eda2a5943e5ecc74c0c5dd" gracePeriod=30
Sep 07 13:21:04.670135 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:04.670095    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"]
Sep 07 13:21:04.670427 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:04.670394    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c" podUID="f18eb88e-3691-43bc-adb0-c0bd0f535a35" containerName="payload-pre-processing" containerID="cri-o://049032e15e91ba8503987265f19dc96fab32e754aeb38f3d0171331dc115b82e" gracePeriod=30
Sep 07 13:21:04.929831 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:04.929799    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"
Sep 07 13:21:04.954614 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:04.954587    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"
Sep 07 13:21:04.993681 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:04.993649    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kzxg\" (UniqueName: \"kubernetes.io/projected/52bc72f1-b2f2-4221-98aa-50e9af10db28-kube-api-access-8kzxg\") pod \"52bc72f1-b2f2-4221-98aa-50e9af10db28\" (UID: \"52bc72f1-b2f2-4221-98aa-50e9af10db28\") "
Sep 07 13:21:04.993870 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:04.993718    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18eb88e-3691-43bc-adb0-c0bd0f535a35-plugins-config-volume\") pod \"f18eb88e-3691-43bc-adb0-c0bd0f535a35\" (UID: \"f18eb88e-3691-43bc-adb0-c0bd0f535a35\") "
Sep 07 13:21:04.993870 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:04.993754    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvz5w\" (UniqueName: \"kubernetes.io/projected/f18eb88e-3691-43bc-adb0-c0bd0f535a35-kube-api-access-xvz5w\") pod \"f18eb88e-3691-43bc-adb0-c0bd0f535a35\" (UID: \"f18eb88e-3691-43bc-adb0-c0bd0f535a35\") "
Sep 07 13:21:04.993870 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:04.993776    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bc72f1-b2f2-4221-98aa-50e9af10db28-plugins-config-volume\") pod \"52bc72f1-b2f2-4221-98aa-50e9af10db28\" (UID: \"52bc72f1-b2f2-4221-98aa-50e9af10db28\") "
Sep 07 13:21:04.994414 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:04.994124    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18eb88e-3691-43bc-adb0-c0bd0f535a35-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f18eb88e-3691-43bc-adb0-c0bd0f535a35" (UID: "f18eb88e-3691-43bc-adb0-c0bd0f535a35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:21:04.994414 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:04.994239    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18eb88e-3691-43bc-adb0-c0bd0f535a35-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:21:04.994768 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:04.994473    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bc72f1-b2f2-4221-98aa-50e9af10db28-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "52bc72f1-b2f2-4221-98aa-50e9af10db28" (UID: "52bc72f1-b2f2-4221-98aa-50e9af10db28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:21:04.996412 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:04.996387    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18eb88e-3691-43bc-adb0-c0bd0f535a35-kube-api-access-xvz5w" (OuterVolumeSpecName: "kube-api-access-xvz5w") pod "f18eb88e-3691-43bc-adb0-c0bd0f535a35" (UID: "f18eb88e-3691-43bc-adb0-c0bd0f535a35"). InnerVolumeSpecName "kube-api-access-xvz5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:21:04.996506 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:04.996420    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bc72f1-b2f2-4221-98aa-50e9af10db28-kube-api-access-8kzxg" (OuterVolumeSpecName: "kube-api-access-8kzxg") pod "52bc72f1-b2f2-4221-98aa-50e9af10db28" (UID: "52bc72f1-b2f2-4221-98aa-50e9af10db28"). InnerVolumeSpecName "kube-api-access-8kzxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:21:05.095075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.095041    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvz5w\" (UniqueName: \"kubernetes.io/projected/f18eb88e-3691-43bc-adb0-c0bd0f535a35-kube-api-access-xvz5w\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:21:05.095075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.095072    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bc72f1-b2f2-4221-98aa-50e9af10db28-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:21:05.095075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.095081    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kzxg\" (UniqueName: \"kubernetes.io/projected/52bc72f1-b2f2-4221-98aa-50e9af10db28-kube-api-access-8kzxg\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:21:05.578997 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.578958    2565 generic.go:356] "Generic (PLEG): container finished" podID="f18eb88e-3691-43bc-adb0-c0bd0f535a35" containerID="049032e15e91ba8503987265f19dc96fab32e754aeb38f3d0171331dc115b82e" exitCode=0
Sep 07 13:21:05.579183 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.579007    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"
Sep 07 13:21:05.579183 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.579047    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c" event={"ID":"f18eb88e-3691-43bc-adb0-c0bd0f535a35","Type":"ContainerDied","Data":"049032e15e91ba8503987265f19dc96fab32e754aeb38f3d0171331dc115b82e"}
Sep 07 13:21:05.579183 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.579108    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c" event={"ID":"f18eb88e-3691-43bc-adb0-c0bd0f535a35","Type":"ContainerDied","Data":"290a6c5eda794a9e103c665cdd2b6fbea8b31a4a84eb0312c7a0387f2df6e383"}
Sep 07 13:21:05.579183 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.579139    2565 scope.go:117] "RemoveContainer" containerID="049032e15e91ba8503987265f19dc96fab32e754aeb38f3d0171331dc115b82e"
Sep 07 13:21:05.580572 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.580546    2565 generic.go:356] "Generic (PLEG): container finished" podID="52bc72f1-b2f2-4221-98aa-50e9af10db28" containerID="4137c757fad7e02bca56884d30c88af9ad6cc9ef10eda2a5943e5ecc74c0c5dd" exitCode=0
Sep 07 13:21:05.580712 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.580576    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw" event={"ID":"52bc72f1-b2f2-4221-98aa-50e9af10db28","Type":"ContainerDied","Data":"4137c757fad7e02bca56884d30c88af9ad6cc9ef10eda2a5943e5ecc74c0c5dd"}
Sep 07 13:21:05.580712 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.580589    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"
Sep 07 13:21:05.580712 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.580619    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw" event={"ID":"52bc72f1-b2f2-4221-98aa-50e9af10db28","Type":"ContainerDied","Data":"7cf7c270303d06701585958a7fda9e34ee2905a83ee589ecf184ddd103d742bf"}
Sep 07 13:21:05.588286 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.588263    2565 scope.go:117] "RemoveContainer" containerID="049032e15e91ba8503987265f19dc96fab32e754aeb38f3d0171331dc115b82e"
Sep 07 13:21:05.588585 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:21:05.588562    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"049032e15e91ba8503987265f19dc96fab32e754aeb38f3d0171331dc115b82e\": container with ID starting with 049032e15e91ba8503987265f19dc96fab32e754aeb38f3d0171331dc115b82e not found: ID does not exist" containerID="049032e15e91ba8503987265f19dc96fab32e754aeb38f3d0171331dc115b82e"
Sep 07 13:21:05.588676 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.588593    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"049032e15e91ba8503987265f19dc96fab32e754aeb38f3d0171331dc115b82e"} err="failed to get container status \"049032e15e91ba8503987265f19dc96fab32e754aeb38f3d0171331dc115b82e\": rpc error: code = NotFound desc = could not find container \"049032e15e91ba8503987265f19dc96fab32e754aeb38f3d0171331dc115b82e\": container with ID starting with 049032e15e91ba8503987265f19dc96fab32e754aeb38f3d0171331dc115b82e not found: ID does not exist"
Sep 07 13:21:05.588676 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.588611    2565 scope.go:117] "RemoveContainer" containerID="4137c757fad7e02bca56884d30c88af9ad6cc9ef10eda2a5943e5ecc74c0c5dd"
Sep 07 13:21:05.596828 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.596811    2565 scope.go:117] "RemoveContainer" containerID="4137c757fad7e02bca56884d30c88af9ad6cc9ef10eda2a5943e5ecc74c0c5dd"
Sep 07 13:21:05.597103 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:21:05.597084    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4137c757fad7e02bca56884d30c88af9ad6cc9ef10eda2a5943e5ecc74c0c5dd\": container with ID starting with 4137c757fad7e02bca56884d30c88af9ad6cc9ef10eda2a5943e5ecc74c0c5dd not found: ID does not exist" containerID="4137c757fad7e02bca56884d30c88af9ad6cc9ef10eda2a5943e5ecc74c0c5dd"
Sep 07 13:21:05.597167 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.597109    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4137c757fad7e02bca56884d30c88af9ad6cc9ef10eda2a5943e5ecc74c0c5dd"} err="failed to get container status \"4137c757fad7e02bca56884d30c88af9ad6cc9ef10eda2a5943e5ecc74c0c5dd\": rpc error: code = NotFound desc = could not find container \"4137c757fad7e02bca56884d30c88af9ad6cc9ef10eda2a5943e5ecc74c0c5dd\": container with ID starting with 4137c757fad7e02bca56884d30c88af9ad6cc9ef10eda2a5943e5ecc74c0c5dd not found: ID does not exist"
Sep 07 13:21:05.598287 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.598267    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"]
Sep 07 13:21:05.603485 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.603461    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ec10ff-7d84797d-7rv4c"]
Sep 07 13:21:05.615621 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.615597    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"]
Sep 07 13:21:05.620304 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.620282    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ec10ff-7b58b5b5f5-dz2jw"]
Sep 07 13:21:05.819976 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:05.819940    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:21:07.488800 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:07.488763    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52bc72f1-b2f2-4221-98aa-50e9af10db28" path="/var/lib/kubelet/pods/52bc72f1-b2f2-4221-98aa-50e9af10db28/volumes"
Sep 07 13:21:07.489181 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:07.489133    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f18eb88e-3691-43bc-adb0-c0bd0f535a35" path="/var/lib/kubelet/pods/f18eb88e-3691-43bc-adb0-c0bd0f535a35/volumes"
Sep 07 13:21:10.261525 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:10.261490    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"]
Sep 07 13:21:15.709684 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:15.709647    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:21:25.282152 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.282045    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7" podUID="ff951c43-9201-447b-af44-1c3dcfae1fd6" containerName="main" containerID="cri-o://5676c46f1b8461d4da6a896d9fd1600e8d6a878c2062c3bc57f73383617fe73d" gracePeriod=45
Sep 07 13:21:25.541081 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.541016    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:21:25.690779 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.690741    2565 generic.go:356] "Generic (PLEG): container finished" podID="ff951c43-9201-447b-af44-1c3dcfae1fd6" containerID="5676c46f1b8461d4da6a896d9fd1600e8d6a878c2062c3bc57f73383617fe73d" exitCode=0
Sep 07 13:21:25.690956 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.690797    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"
Sep 07 13:21:25.690956 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.690827    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7" event={"ID":"ff951c43-9201-447b-af44-1c3dcfae1fd6","Type":"ContainerDied","Data":"5676c46f1b8461d4da6a896d9fd1600e8d6a878c2062c3bc57f73383617fe73d"}
Sep 07 13:21:25.690956 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.690871    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7" event={"ID":"ff951c43-9201-447b-af44-1c3dcfae1fd6","Type":"ContainerDied","Data":"3ab4280febfffd6cbf4d17985ac21a21b954a3c19360774997ba06edf2e613d7"}
Sep 07 13:21:25.690956 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.690890    2565 scope.go:117] "RemoveContainer" containerID="5676c46f1b8461d4da6a896d9fd1600e8d6a878c2062c3bc57f73383617fe73d"
Sep 07 13:21:25.699458 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.699436    2565 scope.go:117] "RemoveContainer" containerID="5676c46f1b8461d4da6a896d9fd1600e8d6a878c2062c3bc57f73383617fe73d"
Sep 07 13:21:25.699742 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:21:25.699722    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5676c46f1b8461d4da6a896d9fd1600e8d6a878c2062c3bc57f73383617fe73d\": container with ID starting with 5676c46f1b8461d4da6a896d9fd1600e8d6a878c2062c3bc57f73383617fe73d not found: ID does not exist" containerID="5676c46f1b8461d4da6a896d9fd1600e8d6a878c2062c3bc57f73383617fe73d"
Sep 07 13:21:25.699797 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.699751    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5676c46f1b8461d4da6a896d9fd1600e8d6a878c2062c3bc57f73383617fe73d"} err="failed to get container status \"5676c46f1b8461d4da6a896d9fd1600e8d6a878c2062c3bc57f73383617fe73d\": rpc error: code = NotFound desc = could not find container \"5676c46f1b8461d4da6a896d9fd1600e8d6a878c2062c3bc57f73383617fe73d\": container with ID starting with 5676c46f1b8461d4da6a896d9fd1600e8d6a878c2062c3bc57f73383617fe73d not found: ID does not exist"
Sep 07 13:21:25.702096 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.702078    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25fs7\" (UniqueName: \"kubernetes.io/projected/ff951c43-9201-447b-af44-1c3dcfae1fd6-kube-api-access-25fs7\") pod \"ff951c43-9201-447b-af44-1c3dcfae1fd6\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") "
Sep 07 13:21:25.702178 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.702143    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff951c43-9201-447b-af44-1c3dcfae1fd6-tls-certs\") pod \"ff951c43-9201-447b-af44-1c3dcfae1fd6\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") "
Sep 07 13:21:25.702178 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.702167    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-home\") pod \"ff951c43-9201-447b-af44-1c3dcfae1fd6\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") "
Sep 07 13:21:25.702292 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.702206    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-model-cache\") pod \"ff951c43-9201-447b-af44-1c3dcfae1fd6\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") "
Sep 07 13:21:25.702292 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.702230    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-dshm\") pod \"ff951c43-9201-447b-af44-1c3dcfae1fd6\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") "
Sep 07 13:21:25.702292 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.702248    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-tmp-dir\") pod \"ff951c43-9201-447b-af44-1c3dcfae1fd6\" (UID: \"ff951c43-9201-447b-af44-1c3dcfae1fd6\") "
Sep 07 13:21:25.702509 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.702469    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-home" (OuterVolumeSpecName: "home") pod "ff951c43-9201-447b-af44-1c3dcfae1fd6" (UID: "ff951c43-9201-447b-af44-1c3dcfae1fd6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:21:25.702599 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.702527    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff951c43-9201-447b-af44-1c3dcfae1fd6" (UID: "ff951c43-9201-447b-af44-1c3dcfae1fd6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:21:25.702660 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.702640    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-home\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:21:25.702660 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.702650    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff951c43-9201-447b-af44-1c3dcfae1fd6" (UID: "ff951c43-9201-447b-af44-1c3dcfae1fd6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:21:25.704483 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.704463    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff951c43-9201-447b-af44-1c3dcfae1fd6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff951c43-9201-447b-af44-1c3dcfae1fd6" (UID: "ff951c43-9201-447b-af44-1c3dcfae1fd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:21:25.704572 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.704487    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-dshm" (OuterVolumeSpecName: "dshm") pod "ff951c43-9201-447b-af44-1c3dcfae1fd6" (UID: "ff951c43-9201-447b-af44-1c3dcfae1fd6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:21:25.704572 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.704525    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff951c43-9201-447b-af44-1c3dcfae1fd6-kube-api-access-25fs7" (OuterVolumeSpecName: "kube-api-access-25fs7") pod "ff951c43-9201-447b-af44-1c3dcfae1fd6" (UID: "ff951c43-9201-447b-af44-1c3dcfae1fd6"). InnerVolumeSpecName "kube-api-access-25fs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:21:25.803998 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.803965    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-model-cache\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:21:25.803998 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.803996    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-dshm\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:21:25.803998 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.804007    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff951c43-9201-447b-af44-1c3dcfae1fd6-tmp-dir\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:21:25.804229 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.804015    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25fs7\" (UniqueName: \"kubernetes.io/projected/ff951c43-9201-447b-af44-1c3dcfae1fd6-kube-api-access-25fs7\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:21:25.804229 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:25.804026    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff951c43-9201-447b-af44-1c3dcfae1fd6-tls-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:21:26.015872 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:26.015839    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"]
Sep 07 13:21:26.025474 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:26.025450    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-ec10ff/ipp-route-ec10ff-kserve-67f6b66d75-5znx7"]
Sep 07 13:21:27.494771 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:21:27.494735    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff951c43-9201-447b-af44-1c3dcfae1fd6" path="/var/lib/kubelet/pods/ff951c43-9201-447b-af44-1c3dcfae1fd6/volumes"
Sep 07 13:22:20.185275 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185235    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn"]
Sep 07 13:22:20.185853 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185727    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1023b67-5901-4482-8886-8912fd062eda" containerName="istio-proxy"
Sep 07 13:22:20.185853 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185740    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1023b67-5901-4482-8886-8912fd062eda" containerName="istio-proxy"
Sep 07 13:22:20.185853 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185751    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff951c43-9201-447b-af44-1c3dcfae1fd6" containerName="main"
Sep 07 13:22:20.185853 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185756    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff951c43-9201-447b-af44-1c3dcfae1fd6" containerName="main"
Sep 07 13:22:20.185853 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185775    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1023b67-5901-4482-8886-8912fd062eda" containerName="istio-proxy"
Sep 07 13:22:20.185853 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185785    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1023b67-5901-4482-8886-8912fd062eda" containerName="istio-proxy"
Sep 07 13:22:20.185853 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185793    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f18eb88e-3691-43bc-adb0-c0bd0f535a35" containerName="payload-pre-processing"
Sep 07 13:22:20.185853 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185798    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f18eb88e-3691-43bc-adb0-c0bd0f535a35" containerName="payload-pre-processing"
Sep 07 13:22:20.185853 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185804    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52bc72f1-b2f2-4221-98aa-50e9af10db28" containerName="payload-processing"
Sep 07 13:22:20.185853 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185809    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="52bc72f1-b2f2-4221-98aa-50e9af10db28" containerName="payload-processing"
Sep 07 13:22:20.185853 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185822    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3fea46d-0b54-4f1a-abbf-4c57878347a4" containerName="revoke-keys"
Sep 07 13:22:20.185853 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185827    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3fea46d-0b54-4f1a-abbf-4c57878347a4" containerName="revoke-keys"
Sep 07 13:22:20.186285 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185906    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1023b67-5901-4482-8886-8912fd062eda" containerName="istio-proxy"
Sep 07 13:22:20.186285 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185919    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3fea46d-0b54-4f1a-abbf-4c57878347a4" containerName="revoke-keys"
Sep 07 13:22:20.186285 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185932    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="52bc72f1-b2f2-4221-98aa-50e9af10db28" containerName="payload-processing"
Sep 07 13:22:20.186285 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185946    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff951c43-9201-447b-af44-1c3dcfae1fd6" containerName="main"
Sep 07 13:22:20.186285 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.185955    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f18eb88e-3691-43bc-adb0-c0bd0f535a35" containerName="payload-pre-processing"
Sep 07 13:22:20.189054 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.189039    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn"
Sep 07 13:22:20.191522 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.191498    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flngs\""
Sep 07 13:22:20.191641 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.191502    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 13:22:20.196751 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.196675    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn"]
Sep 07 13:22:20.242527 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.242487    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d29c38-af04-49b7-bea0-c1caf2accefb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn\" (UID: \"a9d29c38-af04-49b7-bea0-c1caf2accefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn"
Sep 07 13:22:20.343992 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.343960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d29c38-af04-49b7-bea0-c1caf2accefb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn\" (UID: \"a9d29c38-af04-49b7-bea0-c1caf2accefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn"
Sep 07 13:22:20.344581 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.344563    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d29c38-af04-49b7-bea0-c1caf2accefb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn\" (UID: \"a9d29c38-af04-49b7-bea0-c1caf2accefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn"
Sep 07 13:22:20.501024 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.500937    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn"
Sep 07 13:22:20.773728 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.773703    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn"]
Sep 07 13:22:20.776300 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:22:20.776268    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d29c38_af04_49b7_bea0_c1caf2accefb.slice/crio-22bff3a1b112840732f0ecd159c53c04fdb4ae42767e796e3f95dfe1c276b5b3 WatchSource:0}: Error finding container 22bff3a1b112840732f0ecd159c53c04fdb4ae42767e796e3f95dfe1c276b5b3: Status 404 returned error can't find the container with id 22bff3a1b112840732f0ecd159c53c04fdb4ae42767e796e3f95dfe1c276b5b3
Sep 07 13:22:20.986506 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:20.986471    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn" event={"ID":"a9d29c38-af04-49b7-bea0-c1caf2accefb","Type":"ContainerStarted","Data":"22bff3a1b112840732f0ecd159c53c04fdb4ae42767e796e3f95dfe1c276b5b3"}
Sep 07 13:22:21.995189 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:21.995152    2565 generic.go:356] "Generic (PLEG): container finished" podID="a9d29c38-af04-49b7-bea0-c1caf2accefb" containerID="c988191ec85c7fc4e66cbcbb922f27d5b746bd40bdcad62d0af581daa2664e21" exitCode=0
Sep 07 13:22:21.995677 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:21.995208    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn" event={"ID":"a9d29c38-af04-49b7-bea0-c1caf2accefb","Type":"ContainerDied","Data":"c988191ec85c7fc4e66cbcbb922f27d5b746bd40bdcad62d0af581daa2664e21"}
Sep 07 13:22:23.133624 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:23.133600    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn"
Sep 07 13:22:23.170428 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:23.170390    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d29c38-af04-49b7-bea0-c1caf2accefb-maas-api-service-ca\") pod \"a9d29c38-af04-49b7-bea0-c1caf2accefb\" (UID: \"a9d29c38-af04-49b7-bea0-c1caf2accefb\") "
Sep 07 13:22:23.170760 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:23.170726    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d29c38-af04-49b7-bea0-c1caf2accefb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9d29c38-af04-49b7-bea0-c1caf2accefb" (UID: "a9d29c38-af04-49b7-bea0-c1caf2accefb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:22:23.272372 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:23.272249    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d29c38-af04-49b7-bea0-c1caf2accefb-maas-api-service-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:22:24.007430 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:24.007404    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn"
Sep 07 13:22:24.007430 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:24.007418    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn" event={"ID":"a9d29c38-af04-49b7-bea0-c1caf2accefb","Type":"ContainerDied","Data":"22bff3a1b112840732f0ecd159c53c04fdb4ae42767e796e3f95dfe1c276b5b3"}
Sep 07 13:22:24.007637 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:24.007466    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22bff3a1b112840732f0ecd159c53c04fdb4ae42767e796e3f95dfe1c276b5b3"
Sep 07 13:22:30.785288 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:30.785249    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"]
Sep 07 13:22:30.785763 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:30.785514    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj" podUID="fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb" containerName="payload-pre-processing" containerID="cri-o://385c5c51811519797ccd481d224a9a3f5adba6cdd4f52018083f3b1861d6c624" gracePeriod=30
Sep 07 13:22:30.961711 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:30.961678    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn"]
Sep 07 13:22:30.965090 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:30.965064    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-688b44-cbxxn"]
Sep 07 13:22:31.038762 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.038700    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"
Sep 07 13:22:31.044138 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.044109    2565 generic.go:356] "Generic (PLEG): container finished" podID="fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb" containerID="385c5c51811519797ccd481d224a9a3f5adba6cdd4f52018083f3b1861d6c624" exitCode=0
Sep 07 13:22:31.044236 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.044155    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"
Sep 07 13:22:31.044288 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.044232    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj" event={"ID":"fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb","Type":"ContainerDied","Data":"385c5c51811519797ccd481d224a9a3f5adba6cdd4f52018083f3b1861d6c624"}
Sep 07 13:22:31.044288 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.044282    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj" event={"ID":"fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb","Type":"ContainerDied","Data":"776cd10fd647e1498b8a304df17a2c5352af971eefd6eb6eca6fcf8b15a5f5bf"}
Sep 07 13:22:31.044398 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.044304    2565 scope.go:117] "RemoveContainer" containerID="385c5c51811519797ccd481d224a9a3f5adba6cdd4f52018083f3b1861d6c624"
Sep 07 13:22:31.053972 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.053955    2565 scope.go:117] "RemoveContainer" containerID="385c5c51811519797ccd481d224a9a3f5adba6cdd4f52018083f3b1861d6c624"
Sep 07 13:22:31.054301 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:22:31.054279    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385c5c51811519797ccd481d224a9a3f5adba6cdd4f52018083f3b1861d6c624\": container with ID starting with 385c5c51811519797ccd481d224a9a3f5adba6cdd4f52018083f3b1861d6c624 not found: ID does not exist" containerID="385c5c51811519797ccd481d224a9a3f5adba6cdd4f52018083f3b1861d6c624"
Sep 07 13:22:31.054442 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.054307    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385c5c51811519797ccd481d224a9a3f5adba6cdd4f52018083f3b1861d6c624"} err="failed to get container status \"385c5c51811519797ccd481d224a9a3f5adba6cdd4f52018083f3b1861d6c624\": rpc error: code = NotFound desc = could not find container \"385c5c51811519797ccd481d224a9a3f5adba6cdd4f52018083f3b1861d6c624\": container with ID starting with 385c5c51811519797ccd481d224a9a3f5adba6cdd4f52018083f3b1861d6c624 not found: ID does not exist"
Sep 07 13:22:31.146183 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.146153    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzjxr\" (UniqueName: \"kubernetes.io/projected/fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb-kube-api-access-zzjxr\") pod \"fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb\" (UID: \"fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb\") "
Sep 07 13:22:31.146384 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.146257    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb-plugins-config-volume\") pod \"fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb\" (UID: \"fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb\") "
Sep 07 13:22:31.146614 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.146592    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb" (UID: "fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:22:31.148421 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.148384    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb-kube-api-access-zzjxr" (OuterVolumeSpecName: "kube-api-access-zzjxr") pod "fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb" (UID: "fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb"). InnerVolumeSpecName "kube-api-access-zzjxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:22:31.247082 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.247047    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzjxr\" (UniqueName: \"kubernetes.io/projected/fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb-kube-api-access-zzjxr\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:22:31.247082 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.247078    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:22:31.369672 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.369638    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"]
Sep 07 13:22:31.374609 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.374585    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-688b44-584574bb6-jwsdj"]
Sep 07 13:22:31.487765 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.487729    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9d29c38-af04-49b7-bea0-c1caf2accefb" path="/var/lib/kubelet/pods/a9d29c38-af04-49b7-bea0-c1caf2accefb/volumes"
Sep 07 13:22:31.488106 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:31.488093    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb" path="/var/lib/kubelet/pods/fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb/volumes"
Sep 07 13:22:35.230074 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:35.230030    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:22:38.548563 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:38.548528    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"]
Sep 07 13:22:38.548939 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:38.548738    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck" podUID="bd4cbe7f-35b4-45df-92d7-778847de0c25" containerName="istio-proxy" containerID="cri-o://1883dddaf824f900aca364dd8e6ad97a34b505c962b63f8fa97b5269293b512a" gracePeriod=30
Sep 07 13:22:38.977549 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:38.977439    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778"]
Sep 07 13:22:38.977959 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:38.977938    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb" containerName="payload-pre-processing"
Sep 07 13:22:38.978061 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:38.977961    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb" containerName="payload-pre-processing"
Sep 07 13:22:38.978061 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:38.977979    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d29c38-af04-49b7-bea0-c1caf2accefb" containerName="revoke-keys"
Sep 07 13:22:38.978061 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:38.977986    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d29c38-af04-49b7-bea0-c1caf2accefb" containerName="revoke-keys"
Sep 07 13:22:38.978224 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:38.978107    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb2dce2c-b171-40ce-8e2e-0b02cdeb79cb" containerName="payload-pre-processing"
Sep 07 13:22:38.978224 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:38.978124    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9d29c38-af04-49b7-bea0-c1caf2accefb" containerName="revoke-keys"
Sep 07 13:22:38.978224 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:38.978136    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1023b67-5901-4482-8886-8912fd062eda" containerName="istio-proxy"
Sep 07 13:22:38.981344 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:38.981309    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778"
Sep 07 13:22:38.984584 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:38.984560    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 13:22:38.984703 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:38.984577    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flngs\""
Sep 07 13:22:38.993423 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:38.993394    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778"]
Sep 07 13:22:39.126634 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:39.126598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2880a5ed-047d-4864-8d2d-977f367cdff5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778\" (UID: \"2880a5ed-047d-4864-8d2d-977f367cdff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778"
Sep 07 13:22:39.227894 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:39.227798    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2880a5ed-047d-4864-8d2d-977f367cdff5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778\" (UID: \"2880a5ed-047d-4864-8d2d-977f367cdff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778"
Sep 07 13:22:39.228472 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:39.228451    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2880a5ed-047d-4864-8d2d-977f367cdff5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778\" (UID: \"2880a5ed-047d-4864-8d2d-977f367cdff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778"
Sep 07 13:22:39.292426 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:39.292378    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778"
Sep 07 13:22:39.567880 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:39.567854    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778"]
Sep 07 13:22:39.570171 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:22:39.570142    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2880a5ed_047d_4864_8d2d_977f367cdff5.slice/crio-39e467f56496f17a023ad754eae5b87b5f13f4a817a8db6e34d6c13b7145fee2 WatchSource:0}: Error finding container 39e467f56496f17a023ad754eae5b87b5f13f4a817a8db6e34d6c13b7145fee2: Status 404 returned error can't find the container with id 39e467f56496f17a023ad754eae5b87b5f13f4a817a8db6e34d6c13b7145fee2
Sep 07 13:22:40.095551 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:40.095511    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778" event={"ID":"2880a5ed-047d-4864-8d2d-977f367cdff5","Type":"ContainerStarted","Data":"984cbc952005eefc4eda821512ff93939d6841ecb89f91cf7298221b970436a7"}
Sep 07 13:22:40.095733 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:40.095578    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778" event={"ID":"2880a5ed-047d-4864-8d2d-977f367cdff5","Type":"ContainerStarted","Data":"39e467f56496f17a023ad754eae5b87b5f13f4a817a8db6e34d6c13b7145fee2"}
Sep 07 13:22:40.113957 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:40.113907    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778" podStartSLOduration=2.113890583 podStartE2EDuration="2.113890583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:22:40.113373321 +0000 UTC m=+1777.144239632" watchObservedRunningTime="2026-09-07 13:22:40.113890583 +0000 UTC m=+1777.144756893"
Sep 07 13:22:41.102531 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:41.102493    2565 generic.go:356] "Generic (PLEG): container finished" podID="2880a5ed-047d-4864-8d2d-977f367cdff5" containerID="984cbc952005eefc4eda821512ff93939d6841ecb89f91cf7298221b970436a7" exitCode=0
Sep 07 13:22:41.102981 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:41.102563    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778" event={"ID":"2880a5ed-047d-4864-8d2d-977f367cdff5","Type":"ContainerDied","Data":"984cbc952005eefc4eda821512ff93939d6841ecb89f91cf7298221b970436a7"}
Sep 07 13:22:41.790125 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:41.790077    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 13:22:41.790302 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:41.790152    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck" podUID="bd4cbe7f-35b4-45df-92d7-778847de0c25" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 13:22:42.242274 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:42.242245    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778"
Sep 07 13:22:42.357497 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:42.357460    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2880a5ed-047d-4864-8d2d-977f367cdff5-maas-api-service-ca\") pod \"2880a5ed-047d-4864-8d2d-977f367cdff5\" (UID: \"2880a5ed-047d-4864-8d2d-977f367cdff5\") "
Sep 07 13:22:42.357888 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:42.357858    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2880a5ed-047d-4864-8d2d-977f367cdff5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2880a5ed-047d-4864-8d2d-977f367cdff5" (UID: "2880a5ed-047d-4864-8d2d-977f367cdff5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:22:42.459213 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:42.459128    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2880a5ed-047d-4864-8d2d-977f367cdff5-maas-api-service-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:22:43.115049 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.115017    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778"
Sep 07 13:22:43.115225 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.115043    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778" event={"ID":"2880a5ed-047d-4864-8d2d-977f367cdff5","Type":"ContainerDied","Data":"39e467f56496f17a023ad754eae5b87b5f13f4a817a8db6e34d6c13b7145fee2"}
Sep 07 13:22:43.115225 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.115115    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39e467f56496f17a023ad754eae5b87b5f13f4a817a8db6e34d6c13b7145fee2"
Sep 07 13:22:43.799915 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.799889    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:22:43.975070 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.974971    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-credential-socket\") pod \"bd4cbe7f-35b4-45df-92d7-778847de0c25\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") "
Sep 07 13:22:43.975246 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.975071    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-podinfo\") pod \"bd4cbe7f-35b4-45df-92d7-778847de0c25\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") "
Sep 07 13:22:43.975246 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.975110    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87ft8\" (UniqueName: \"kubernetes.io/projected/bd4cbe7f-35b4-45df-92d7-778847de0c25-kube-api-access-87ft8\") pod \"bd4cbe7f-35b4-45df-92d7-778847de0c25\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") "
Sep 07 13:22:43.975246 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.975137    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-workload-socket\") pod \"bd4cbe7f-35b4-45df-92d7-778847de0c25\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") "
Sep 07 13:22:43.975246 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.975190    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-workload-certs\") pod \"bd4cbe7f-35b4-45df-92d7-778847de0c25\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") "
Sep 07 13:22:43.975246 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.975227    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-envoy\") pod \"bd4cbe7f-35b4-45df-92d7-778847de0c25\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") "
Sep 07 13:22:43.975246 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.975232    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bd4cbe7f-35b4-45df-92d7-778847de0c25" (UID: "bd4cbe7f-35b4-45df-92d7-778847de0c25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:22:43.975631 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.975263    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd4cbe7f-35b4-45df-92d7-778847de0c25-istiod-ca-cert\") pod \"bd4cbe7f-35b4-45df-92d7-778847de0c25\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") "
Sep 07 13:22:43.975631 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.975300    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-data\") pod \"bd4cbe7f-35b4-45df-92d7-778847de0c25\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") "
Sep 07 13:22:43.975631 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.975383    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-token\") pod \"bd4cbe7f-35b4-45df-92d7-778847de0c25\" (UID: \"bd4cbe7f-35b4-45df-92d7-778847de0c25\") "
Sep 07 13:22:43.975631 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.975473    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bd4cbe7f-35b4-45df-92d7-778847de0c25" (UID: "bd4cbe7f-35b4-45df-92d7-778847de0c25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:22:43.975631 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.975491    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bd4cbe7f-35b4-45df-92d7-778847de0c25" (UID: "bd4cbe7f-35b4-45df-92d7-778847de0c25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:22:43.975890 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.975773    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-workload-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:22:43.975930 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.975865    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-workload-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:22:43.975982 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.975966    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-credential-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:22:43.976419 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.976387    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-data" (OuterVolumeSpecName: "istio-data") pod "bd4cbe7f-35b4-45df-92d7-778847de0c25" (UID: "bd4cbe7f-35b4-45df-92d7-778847de0c25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:22:43.976566 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.976413    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd4cbe7f-35b4-45df-92d7-778847de0c25-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bd4cbe7f-35b4-45df-92d7-778847de0c25" (UID: "bd4cbe7f-35b4-45df-92d7-778847de0c25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:22:43.977801 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.977774    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bd4cbe7f-35b4-45df-92d7-778847de0c25" (UID: "bd4cbe7f-35b4-45df-92d7-778847de0c25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:22:43.977801 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.977792    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bd4cbe7f-35b4-45df-92d7-778847de0c25" (UID: "bd4cbe7f-35b4-45df-92d7-778847de0c25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 13:22:43.978249 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.978223    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4cbe7f-35b4-45df-92d7-778847de0c25-kube-api-access-87ft8" (OuterVolumeSpecName: "kube-api-access-87ft8") pod "bd4cbe7f-35b4-45df-92d7-778847de0c25" (UID: "bd4cbe7f-35b4-45df-92d7-778847de0c25"). InnerVolumeSpecName "kube-api-access-87ft8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:22:43.978366 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:43.978282    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-token" (OuterVolumeSpecName: "istio-token") pod "bd4cbe7f-35b4-45df-92d7-778847de0c25" (UID: "bd4cbe7f-35b4-45df-92d7-778847de0c25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:22:44.076579 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.076538    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-envoy\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:22:44.076579 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.076570    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd4cbe7f-35b4-45df-92d7-778847de0c25-istiod-ca-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:22:44.076579 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.076579    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-data\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:22:44.076579 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.076586    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-token\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:22:44.076863 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.076595    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd4cbe7f-35b4-45df-92d7-778847de0c25-istio-podinfo\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:22:44.076863 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.076603    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87ft8\" (UniqueName: \"kubernetes.io/projected/bd4cbe7f-35b4-45df-92d7-778847de0c25-kube-api-access-87ft8\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:22:44.122107 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.122072    2565 generic.go:356] "Generic (PLEG): container finished" podID="bd4cbe7f-35b4-45df-92d7-778847de0c25" containerID="1883dddaf824f900aca364dd8e6ad97a34b505c962b63f8fa97b5269293b512a" exitCode=0
Sep 07 13:22:44.122278 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.122123    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"
Sep 07 13:22:44.122278 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.122137    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck" event={"ID":"bd4cbe7f-35b4-45df-92d7-778847de0c25","Type":"ContainerDied","Data":"1883dddaf824f900aca364dd8e6ad97a34b505c962b63f8fa97b5269293b512a"}
Sep 07 13:22:44.122278 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.122191    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck" event={"ID":"bd4cbe7f-35b4-45df-92d7-778847de0c25","Type":"ContainerDied","Data":"819ebc3333550500b3c4e782dd191e1bbb833e169c0fdadbd3b9b55686faf9d2"}
Sep 07 13:22:44.122278 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.122221    2565 scope.go:117] "RemoveContainer" containerID="1883dddaf824f900aca364dd8e6ad97a34b505c962b63f8fa97b5269293b512a"
Sep 07 13:22:44.131845 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.131827    2565 scope.go:117] "RemoveContainer" containerID="ac251a90c03a7f93e7025ab8e50c1341385294250fc75e07ae9721733db6f504"
Sep 07 13:22:44.140417 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.140398    2565 scope.go:117] "RemoveContainer" containerID="1883dddaf824f900aca364dd8e6ad97a34b505c962b63f8fa97b5269293b512a"
Sep 07 13:22:44.140672 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:22:44.140653    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1883dddaf824f900aca364dd8e6ad97a34b505c962b63f8fa97b5269293b512a\": container with ID starting with 1883dddaf824f900aca364dd8e6ad97a34b505c962b63f8fa97b5269293b512a not found: ID does not exist" containerID="1883dddaf824f900aca364dd8e6ad97a34b505c962b63f8fa97b5269293b512a"
Sep 07 13:22:44.140713 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.140681    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1883dddaf824f900aca364dd8e6ad97a34b505c962b63f8fa97b5269293b512a"} err="failed to get container status \"1883dddaf824f900aca364dd8e6ad97a34b505c962b63f8fa97b5269293b512a\": rpc error: code = NotFound desc = could not find container \"1883dddaf824f900aca364dd8e6ad97a34b505c962b63f8fa97b5269293b512a\": container with ID starting with 1883dddaf824f900aca364dd8e6ad97a34b505c962b63f8fa97b5269293b512a not found: ID does not exist"
Sep 07 13:22:44.140713 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.140698    2565 scope.go:117] "RemoveContainer" containerID="ac251a90c03a7f93e7025ab8e50c1341385294250fc75e07ae9721733db6f504"
Sep 07 13:22:44.140966 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:22:44.140947    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac251a90c03a7f93e7025ab8e50c1341385294250fc75e07ae9721733db6f504\": container with ID starting with ac251a90c03a7f93e7025ab8e50c1341385294250fc75e07ae9721733db6f504 not found: ID does not exist" containerID="ac251a90c03a7f93e7025ab8e50c1341385294250fc75e07ae9721733db6f504"
Sep 07 13:22:44.141027 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.140972    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac251a90c03a7f93e7025ab8e50c1341385294250fc75e07ae9721733db6f504"} err="failed to get container status \"ac251a90c03a7f93e7025ab8e50c1341385294250fc75e07ae9721733db6f504\": rpc error: code = NotFound desc = could not find container \"ac251a90c03a7f93e7025ab8e50c1341385294250fc75e07ae9721733db6f504\": container with ID starting with ac251a90c03a7f93e7025ab8e50c1341385294250fc75e07ae9721733db6f504 not found: ID does not exist"
Sep 07 13:22:44.151832 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.151805    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"]
Sep 07 13:22:44.156220 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:44.156194    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-688b44-openshift-default-86f67569dd-qgxck"]
Sep 07 13:22:45.488393 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:45.488359    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd4cbe7f-35b4-45df-92d7-778847de0c25" path="/var/lib/kubelet/pods/bd4cbe7f-35b4-45df-92d7-778847de0c25/volumes"
Sep 07 13:22:49.617456 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:49.617422    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aa813e-29813115-x5dwj"]
Sep 07 13:22:49.621989 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:49.621957    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-aa813e-29813115-x5dwj"]
Sep 07 13:22:49.649739 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:49.649700    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"]
Sep 07 13:22:49.650226 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:49.649938    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh" podUID="6e21fb80-e2d9-425e-b336-449536271a12" containerName="payload-processing" containerID="cri-o://5ce433d25d54d5d5a851e12ff68b484aba68a447e73016941cfc9e321c3dc347" gracePeriod=30
Sep 07 13:22:49.837480 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:49.837450    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778"]
Sep 07 13:22:49.843583 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:49.843543    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-aa813e-wf778"]
Sep 07 13:22:49.897879 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:49.897850    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"
Sep 07 13:22:49.933807 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:49.933767    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e21fb80-e2d9-425e-b336-449536271a12-plugins-config-volume\") pod \"6e21fb80-e2d9-425e-b336-449536271a12\" (UID: \"6e21fb80-e2d9-425e-b336-449536271a12\") "
Sep 07 13:22:49.933985 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:49.933903    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh9sl\" (UniqueName: \"kubernetes.io/projected/6e21fb80-e2d9-425e-b336-449536271a12-kube-api-access-xh9sl\") pod \"6e21fb80-e2d9-425e-b336-449536271a12\" (UID: \"6e21fb80-e2d9-425e-b336-449536271a12\") "
Sep 07 13:22:49.934121 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:49.934097    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e21fb80-e2d9-425e-b336-449536271a12-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e21fb80-e2d9-425e-b336-449536271a12" (UID: "6e21fb80-e2d9-425e-b336-449536271a12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:22:49.936031 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:49.936007    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e21fb80-e2d9-425e-b336-449536271a12-kube-api-access-xh9sl" (OuterVolumeSpecName: "kube-api-access-xh9sl") pod "6e21fb80-e2d9-425e-b336-449536271a12" (UID: "6e21fb80-e2d9-425e-b336-449536271a12"). InnerVolumeSpecName "kube-api-access-xh9sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:22:50.035036 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:50.034995    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e21fb80-e2d9-425e-b336-449536271a12-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:22:50.035036 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:50.035028    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh9sl\" (UniqueName: \"kubernetes.io/projected/6e21fb80-e2d9-425e-b336-449536271a12-kube-api-access-xh9sl\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:22:50.154018 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:50.153933    2565 generic.go:356] "Generic (PLEG): container finished" podID="6e21fb80-e2d9-425e-b336-449536271a12" containerID="5ce433d25d54d5d5a851e12ff68b484aba68a447e73016941cfc9e321c3dc347" exitCode=0
Sep 07 13:22:50.154018 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:50.153975    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"
Sep 07 13:22:50.154197 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:50.153977    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh" event={"ID":"6e21fb80-e2d9-425e-b336-449536271a12","Type":"ContainerDied","Data":"5ce433d25d54d5d5a851e12ff68b484aba68a447e73016941cfc9e321c3dc347"}
Sep 07 13:22:50.154197 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:50.154111    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh" event={"ID":"6e21fb80-e2d9-425e-b336-449536271a12","Type":"ContainerDied","Data":"eebc5c9d728d5bd9463187d08716de3c0868ace79397bd323c5c636064ab8908"}
Sep 07 13:22:50.154197 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:50.154139    2565 scope.go:117] "RemoveContainer" containerID="5ce433d25d54d5d5a851e12ff68b484aba68a447e73016941cfc9e321c3dc347"
Sep 07 13:22:50.163462 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:50.163443    2565 scope.go:117] "RemoveContainer" containerID="5ce433d25d54d5d5a851e12ff68b484aba68a447e73016941cfc9e321c3dc347"
Sep 07 13:22:50.163721 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:22:50.163703    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ce433d25d54d5d5a851e12ff68b484aba68a447e73016941cfc9e321c3dc347\": container with ID starting with 5ce433d25d54d5d5a851e12ff68b484aba68a447e73016941cfc9e321c3dc347 not found: ID does not exist" containerID="5ce433d25d54d5d5a851e12ff68b484aba68a447e73016941cfc9e321c3dc347"
Sep 07 13:22:50.163768 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:50.163730    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce433d25d54d5d5a851e12ff68b484aba68a447e73016941cfc9e321c3dc347"} err="failed to get container status \"5ce433d25d54d5d5a851e12ff68b484aba68a447e73016941cfc9e321c3dc347\": rpc error: code = NotFound desc = could not find container \"5ce433d25d54d5d5a851e12ff68b484aba68a447e73016941cfc9e321c3dc347\": container with ID starting with 5ce433d25d54d5d5a851e12ff68b484aba68a447e73016941cfc9e321c3dc347 not found: ID does not exist"
Sep 07 13:22:50.176915 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:50.176893    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"]
Sep 07 13:22:50.181929 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:50.181908    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-aa813e-74f688c85c-lzljh"]
Sep 07 13:22:51.488643 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:51.488606    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2880a5ed-047d-4864-8d2d-977f367cdff5" path="/var/lib/kubelet/pods/2880a5ed-047d-4864-8d2d-977f367cdff5/volumes"
Sep 07 13:22:51.489117 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:51.488952    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e21fb80-e2d9-425e-b336-449536271a12" path="/var/lib/kubelet/pods/6e21fb80-e2d9-425e-b336-449536271a12/volumes"
Sep 07 13:22:51.489198 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:51.489185    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="961bd92b-ace0-488c-8b85-c3eaeb62f239" path="/var/lib/kubelet/pods/961bd92b-ace0-488c-8b85-c3eaeb62f239/volumes"
Sep 07 13:22:53.208338 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:53.208301    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:22:57.575862 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.575775    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67"]
Sep 07 13:22:57.576261 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.576241    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd4cbe7f-35b4-45df-92d7-778847de0c25" containerName="istio-proxy"
Sep 07 13:22:57.576316 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.576266    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd4cbe7f-35b4-45df-92d7-778847de0c25" containerName="istio-proxy"
Sep 07 13:22:57.576316 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.576286    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2880a5ed-047d-4864-8d2d-977f367cdff5" containerName="revoke-keys"
Sep 07 13:22:57.576316 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.576294    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2880a5ed-047d-4864-8d2d-977f367cdff5" containerName="revoke-keys"
Sep 07 13:22:57.576316 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.576307    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e21fb80-e2d9-425e-b336-449536271a12" containerName="payload-processing"
Sep 07 13:22:57.576316 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.576313    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e21fb80-e2d9-425e-b336-449536271a12" containerName="payload-processing"
Sep 07 13:22:57.576519 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.576413    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2880a5ed-047d-4864-8d2d-977f367cdff5" containerName="revoke-keys"
Sep 07 13:22:57.576519 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.576422    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd4cbe7f-35b4-45df-92d7-778847de0c25" containerName="istio-proxy"
Sep 07 13:22:57.576519 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.576431    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e21fb80-e2d9-425e-b336-449536271a12" containerName="payload-processing"
Sep 07 13:22:57.576519 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.576439    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd4cbe7f-35b4-45df-92d7-778847de0c25" containerName="istio-proxy"
Sep 07 13:22:57.580722 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.580704    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67"
Sep 07 13:22:57.583156 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.583124    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 13:22:57.583286 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.583162    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flngs\""
Sep 07 13:22:57.589430 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.589396    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67"]
Sep 07 13:22:57.608604 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.608571    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3beab4a-aebe-49b6-880a-cadb0388acae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67\" (UID: \"b3beab4a-aebe-49b6-880a-cadb0388acae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67"
Sep 07 13:22:57.709801 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.709766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3beab4a-aebe-49b6-880a-cadb0388acae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67\" (UID: \"b3beab4a-aebe-49b6-880a-cadb0388acae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67"
Sep 07 13:22:57.710405 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.710385    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3beab4a-aebe-49b6-880a-cadb0388acae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67\" (UID: \"b3beab4a-aebe-49b6-880a-cadb0388acae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67"
Sep 07 13:22:57.892112 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:57.892028    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67"
Sep 07 13:22:58.167078 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:58.166997    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67"]
Sep 07 13:22:58.170014 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:22:58.169988    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3beab4a_aebe_49b6_880a_cadb0388acae.slice/crio-237ddd17339dd7d7eccbfb878d05f38ad22ad274cf3fde697b0e9a0cf42ae1ec WatchSource:0}: Error finding container 237ddd17339dd7d7eccbfb878d05f38ad22ad274cf3fde697b0e9a0cf42ae1ec: Status 404 returned error can't find the container with id 237ddd17339dd7d7eccbfb878d05f38ad22ad274cf3fde697b0e9a0cf42ae1ec
Sep 07 13:22:58.199535 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:58.199509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67" event={"ID":"b3beab4a-aebe-49b6-880a-cadb0388acae","Type":"ContainerStarted","Data":"237ddd17339dd7d7eccbfb878d05f38ad22ad274cf3fde697b0e9a0cf42ae1ec"}
Sep 07 13:22:59.206593 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:59.206494    2565 generic.go:356] "Generic (PLEG): container finished" podID="b3beab4a-aebe-49b6-880a-cadb0388acae" containerID="dee3659bd996f38dd76e53158119ffdf95f51c2367b2f482f522fde003bb0ed3" exitCode=0
Sep 07 13:22:59.207001 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:22:59.206584    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67" event={"ID":"b3beab4a-aebe-49b6-880a-cadb0388acae","Type":"ContainerDied","Data":"dee3659bd996f38dd76e53158119ffdf95f51c2367b2f482f522fde003bb0ed3"}
Sep 07 13:23:00.342108 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:00.342074    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67"
Sep 07 13:23:00.434385 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:00.434311    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3beab4a-aebe-49b6-880a-cadb0388acae-maas-api-service-ca\") pod \"b3beab4a-aebe-49b6-880a-cadb0388acae\" (UID: \"b3beab4a-aebe-49b6-880a-cadb0388acae\") "
Sep 07 13:23:00.434738 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:00.434704    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3beab4a-aebe-49b6-880a-cadb0388acae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3beab4a-aebe-49b6-880a-cadb0388acae" (UID: "b3beab4a-aebe-49b6-880a-cadb0388acae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:23:00.536101 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:00.536017    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3beab4a-aebe-49b6-880a-cadb0388acae-maas-api-service-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:23:01.217954 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:01.217926    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67"
Sep 07 13:23:01.218148 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:01.217953    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67" event={"ID":"b3beab4a-aebe-49b6-880a-cadb0388acae","Type":"ContainerDied","Data":"237ddd17339dd7d7eccbfb878d05f38ad22ad274cf3fde697b0e9a0cf42ae1ec"}
Sep 07 13:23:01.218148 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:01.217996    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="237ddd17339dd7d7eccbfb878d05f38ad22ad274cf3fde697b0e9a0cf42ae1ec"
Sep 07 13:23:03.824290 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:03.824260    2565 scope.go:117] "RemoveContainer" containerID="4e9fb47ac6102a15a2d9bb22fe3acf004222c161e1169d8f2be0317d8f7b56f4"
Sep 07 13:23:08.173868 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:08.173830    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"]
Sep 07 13:23:08.174283 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:08.174078    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh" podUID="fc5708f7-7362-4455-9d1f-93336590f962" containerName="payload-pre-processing" containerID="cri-o://1d469f63b7f70d5c97abe38c6bd173ecbece40acb980fcacf5c0091c4ce904af" gracePeriod=30
Sep 07 13:23:08.385402 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:08.385365    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67"]
Sep 07 13:23:08.393844 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:08.393798    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae93b0-bxn67"]
Sep 07 13:23:08.427311 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:08.427250    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"
Sep 07 13:23:08.516085 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:08.516052    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dxhb\" (UniqueName: \"kubernetes.io/projected/fc5708f7-7362-4455-9d1f-93336590f962-kube-api-access-5dxhb\") pod \"fc5708f7-7362-4455-9d1f-93336590f962\" (UID: \"fc5708f7-7362-4455-9d1f-93336590f962\") "
Sep 07 13:23:08.516260 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:08.516104    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5708f7-7362-4455-9d1f-93336590f962-plugins-config-volume\") pod \"fc5708f7-7362-4455-9d1f-93336590f962\" (UID: \"fc5708f7-7362-4455-9d1f-93336590f962\") "
Sep 07 13:23:08.516768 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:08.516734    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5708f7-7362-4455-9d1f-93336590f962-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc5708f7-7362-4455-9d1f-93336590f962" (UID: "fc5708f7-7362-4455-9d1f-93336590f962"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:23:08.518317 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:08.518280    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5708f7-7362-4455-9d1f-93336590f962-kube-api-access-5dxhb" (OuterVolumeSpecName: "kube-api-access-5dxhb") pod "fc5708f7-7362-4455-9d1f-93336590f962" (UID: "fc5708f7-7362-4455-9d1f-93336590f962"). InnerVolumeSpecName "kube-api-access-5dxhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:23:08.617044 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:08.617005    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dxhb\" (UniqueName: \"kubernetes.io/projected/fc5708f7-7362-4455-9d1f-93336590f962-kube-api-access-5dxhb\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:23:08.617044 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:08.617037    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5708f7-7362-4455-9d1f-93336590f962-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:23:09.260338 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:09.260273    2565 generic.go:356] "Generic (PLEG): container finished" podID="fc5708f7-7362-4455-9d1f-93336590f962" containerID="1d469f63b7f70d5c97abe38c6bd173ecbece40acb980fcacf5c0091c4ce904af" exitCode=0
Sep 07 13:23:09.260785 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:09.260351    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"
Sep 07 13:23:09.260785 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:09.260362    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh" event={"ID":"fc5708f7-7362-4455-9d1f-93336590f962","Type":"ContainerDied","Data":"1d469f63b7f70d5c97abe38c6bd173ecbece40acb980fcacf5c0091c4ce904af"}
Sep 07 13:23:09.260785 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:09.260423    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh" event={"ID":"fc5708f7-7362-4455-9d1f-93336590f962","Type":"ContainerDied","Data":"132865436ba6df27a12e46a268751cf569e2b65d8822279d0d29c36df0c9445c"}
Sep 07 13:23:09.260785 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:09.260445    2565 scope.go:117] "RemoveContainer" containerID="1d469f63b7f70d5c97abe38c6bd173ecbece40acb980fcacf5c0091c4ce904af"
Sep 07 13:23:09.270130 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:09.270111    2565 scope.go:117] "RemoveContainer" containerID="1d469f63b7f70d5c97abe38c6bd173ecbece40acb980fcacf5c0091c4ce904af"
Sep 07 13:23:09.270418 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:23:09.270397    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d469f63b7f70d5c97abe38c6bd173ecbece40acb980fcacf5c0091c4ce904af\": container with ID starting with 1d469f63b7f70d5c97abe38c6bd173ecbece40acb980fcacf5c0091c4ce904af not found: ID does not exist" containerID="1d469f63b7f70d5c97abe38c6bd173ecbece40acb980fcacf5c0091c4ce904af"
Sep 07 13:23:09.270490 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:09.270425    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d469f63b7f70d5c97abe38c6bd173ecbece40acb980fcacf5c0091c4ce904af"} err="failed to get container status \"1d469f63b7f70d5c97abe38c6bd173ecbece40acb980fcacf5c0091c4ce904af\": rpc error: code = NotFound desc = could not find container \"1d469f63b7f70d5c97abe38c6bd173ecbece40acb980fcacf5c0091c4ce904af\": container with ID starting with 1d469f63b7f70d5c97abe38c6bd173ecbece40acb980fcacf5c0091c4ce904af not found: ID does not exist"
Sep 07 13:23:09.284476 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:09.284442    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"]
Sep 07 13:23:09.289072 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:09.289046    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae93b0-5d4cdc645d-rjmwh"]
Sep 07 13:23:09.488881 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:09.488849    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3beab4a-aebe-49b6-880a-cadb0388acae" path="/var/lib/kubelet/pods/b3beab4a-aebe-49b6-880a-cadb0388acae/volumes"
Sep 07 13:23:09.489220 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:09.489208    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc5708f7-7362-4455-9d1f-93336590f962" path="/var/lib/kubelet/pods/fc5708f7-7362-4455-9d1f-93336590f962/volumes"
Sep 07 13:23:11.013545 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:11.013507    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:23:22.107091 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:23:22.107054    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:24:43.711418 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:24:43.711377    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:25:12.011954 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:25:12.011922    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:25:19.223949 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:25:19.223910    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:25:29.660933 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:25:29.660897    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:25:51.622661 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:25:51.622625    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:25:57.915144 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:25:57.915054    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:26:03.874961 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:26:03.874926    2565 scope.go:117] "RemoveContainer" containerID="3fc87abb0b6fd3fb452b96db21e3a5861d8f7f519043c86272af7c602cf90d2e"
Sep 07 13:27:03.895118 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:03.895087    2565 scope.go:117] "RemoveContainer" containerID="2ab3189dfb5487808aeb0b8fff8281694eaffee2a4539245a79b703d5fdd244f"
Sep 07 13:27:23.919660 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:23.919617    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:27:31.938689 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:31.938599    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:27:40.229430 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:40.229384    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:27:45.732818 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.732784    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"]
Sep 07 13:27:45.733222 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.733169    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc5708f7-7362-4455-9d1f-93336590f962" containerName="payload-pre-processing"
Sep 07 13:27:45.733222 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.733182    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc5708f7-7362-4455-9d1f-93336590f962" containerName="payload-pre-processing"
Sep 07 13:27:45.733222 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.733200    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3beab4a-aebe-49b6-880a-cadb0388acae" containerName="revoke-keys"
Sep 07 13:27:45.733222 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.733206    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3beab4a-aebe-49b6-880a-cadb0388acae" containerName="revoke-keys"
Sep 07 13:27:45.733222 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.733217    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd4cbe7f-35b4-45df-92d7-778847de0c25" containerName="istio-proxy"
Sep 07 13:27:45.733222 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.733224    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd4cbe7f-35b4-45df-92d7-778847de0c25" containerName="istio-proxy"
Sep 07 13:27:45.733476 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.733277    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3beab4a-aebe-49b6-880a-cadb0388acae" containerName="revoke-keys"
Sep 07 13:27:45.733476 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.733290    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc5708f7-7362-4455-9d1f-93336590f962" containerName="payload-pre-processing"
Sep 07 13:27:45.735913 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.735897    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.738667 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.738631    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-580612d9-kserve-self-signed-certs\""
Sep 07 13:27:45.750987 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.750962    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"]
Sep 07 13:27:45.817799 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.817766    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-dshm\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.817972 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.817824    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-model-cache\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.817972 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.817842    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f2c92c0-79d9-447c-98c6-d7902962fffa-tls-certs\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.817972 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.817891    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-tmp-dir\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.817972 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.817958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5kxr\" (UniqueName: \"kubernetes.io/projected/0f2c92c0-79d9-447c-98c6-d7902962fffa-kube-api-access-q5kxr\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.818121 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.817996    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-home\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.919283 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.919237    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-dshm\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.919521 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.919317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-model-cache\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.919521 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.919378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f2c92c0-79d9-447c-98c6-d7902962fffa-tls-certs\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.919521 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.919418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-tmp-dir\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.919521 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.919480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5kxr\" (UniqueName: \"kubernetes.io/projected/0f2c92c0-79d9-447c-98c6-d7902962fffa-kube-api-access-q5kxr\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.919521 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.919516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-home\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.919809 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.919755    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-model-cache\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.919869 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.919814    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-tmp-dir\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.919925 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.919873    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-home\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.921652 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.921623    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-dshm\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.921913 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.921895    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f2c92c0-79d9-447c-98c6-d7902962fffa-tls-certs\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:45.930689 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:45.930662    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5kxr\" (UniqueName: \"kubernetes.io/projected/0f2c92c0-79d9-447c-98c6-d7902962fffa-kube-api-access-q5kxr\") pod \"e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") " pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:46.046798 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:46.046760    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:46.323089 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:46.323060    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"]
Sep 07 13:27:46.325087 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:27:46.325063    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f2c92c0_79d9_447c_98c6_d7902962fffa.slice/crio-109e88b9772bc6caa780ce404714247953762499f37abb74b73a9c4cac613f79 WatchSource:0}: Error finding container 109e88b9772bc6caa780ce404714247953762499f37abb74b73a9c4cac613f79: Status 404 returned error can't find the container with id 109e88b9772bc6caa780ce404714247953762499f37abb74b73a9c4cac613f79
Sep 07 13:27:46.697391 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:46.697288    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh" event={"ID":"0f2c92c0-79d9-447c-98c6-d7902962fffa","Type":"ContainerStarted","Data":"4bb0dc417dbbd121322901b7a105e6834e0d900d63a702e0e6e07f3fd43f7659"}
Sep 07 13:27:46.697391 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:46.697353    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh" event={"ID":"0f2c92c0-79d9-447c-98c6-d7902962fffa","Type":"ContainerStarted","Data":"109e88b9772bc6caa780ce404714247953762499f37abb74b73a9c4cac613f79"}
Sep 07 13:27:46.720136 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:46.720084    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh" podStartSLOduration=1.7200675460000001 podStartE2EDuration="1.720067546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:27:46.718611235 +0000 UTC m=+2083.749477568" watchObservedRunningTime="2026-09-07 13:27:46.720067546 +0000 UTC m=+2083.750933856"
Sep 07 13:27:48.449704 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:48.449666    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:27:56.047161 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:56.047123    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:56.047732 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:56.047179    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:56.059907 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:56.059880    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:56.765896 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:56.765861    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:27:58.033429 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.033397    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"]
Sep 07 13:27:58.037086 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.037066    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.042697 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.042671    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-580612d9-kserve-self-signed-certs\""
Sep 07 13:27:58.055883 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.055854    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"]
Sep 07 13:27:58.147515 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.147476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-home\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.147515 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.147517    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-dshm\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.147721 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.147589    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-model-cache\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.147721 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.147620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-tls-certs\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.147721 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.147644    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-tmp-dir\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.147721 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.147665    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8mwn\" (UniqueName: \"kubernetes.io/projected/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-kube-api-access-n8mwn\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.248788 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.248746    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-model-cache\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.248788 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.248791    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-tls-certs\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.249062 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.248809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-tmp-dir\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.249062 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.248829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8mwn\" (UniqueName: \"kubernetes.io/projected/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-kube-api-access-n8mwn\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.249062 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.248903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-home\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.249062 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.248924    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-dshm\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.249289 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.249203    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-tmp-dir\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.249289 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.249247    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-model-cache\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.249417 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.249300    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-home\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.251377 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.251348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-dshm\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.251530 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.251512    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-tls-certs\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.262721 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.262699    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8mwn\" (UniqueName: \"kubernetes.io/projected/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-kube-api-access-n8mwn\") pod \"e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") " pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.347619 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.347587    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:27:58.629951 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.629924    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"]
Sep 07 13:27:58.631222 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:27:58.631196    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb256e91a_64d6_43bb_8af6_d561ef1d4cf4.slice/crio-01bb007e769de2686d52f9c35640e306811f106e070f7f020807f6b60ada1bbb WatchSource:0}: Error finding container 01bb007e769de2686d52f9c35640e306811f106e070f7f020807f6b60ada1bbb: Status 404 returned error can't find the container with id 01bb007e769de2686d52f9c35640e306811f106e070f7f020807f6b60ada1bbb
Sep 07 13:27:58.766075 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.766044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx" event={"ID":"b256e91a-64d6-43bb-8af6-d561ef1d4cf4","Type":"ContainerStarted","Data":"aa293c2c9a8060d59fb173af21555d6c0d221db53881a4ed59c17a6b510b0030"}
Sep 07 13:27:58.766314 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.766085    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx" event={"ID":"b256e91a-64d6-43bb-8af6-d561ef1d4cf4","Type":"ContainerStarted","Data":"01bb007e769de2686d52f9c35640e306811f106e070f7f020807f6b60ada1bbb"}
Sep 07 13:27:58.795021 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:27:58.794970    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx" podStartSLOduration=0.794953226 podStartE2EDuration="794.953226ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:27:58.79411613 +0000 UTC m=+2095.824982440" watchObservedRunningTime="2026-09-07 13:27:58.794953226 +0000 UTC m=+2095.825819535"
Sep 07 13:28:01.175562 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:01.175522    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:28:08.348022 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:08.347979    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:28:08.348447 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:08.348046    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:28:08.360391 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:08.360370    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:28:08.832418 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:08.832384    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:28:30.282873 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:30.282840    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"]
Sep 07 13:28:31.232619 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:31.232574    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"]
Sep 07 13:28:33.426489 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:33.426452    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:28:41.009026 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:41.008939    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:28:45.302882 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.302838    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh" podUID="0f2c92c0-79d9-447c-98c6-d7902962fffa" containerName="main" containerID="cri-o://4bb0dc417dbbd121322901b7a105e6834e0d900d63a702e0e6e07f3fd43f7659" gracePeriod=45
Sep 07 13:28:45.556009 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.555946    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:28:45.611426 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.611394    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-tmp-dir\") pod \"0f2c92c0-79d9-447c-98c6-d7902962fffa\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") "
Sep 07 13:28:45.611597 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.611436    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-dshm\") pod \"0f2c92c0-79d9-447c-98c6-d7902962fffa\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") "
Sep 07 13:28:45.611597 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.611504    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5kxr\" (UniqueName: \"kubernetes.io/projected/0f2c92c0-79d9-447c-98c6-d7902962fffa-kube-api-access-q5kxr\") pod \"0f2c92c0-79d9-447c-98c6-d7902962fffa\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") "
Sep 07 13:28:45.611597 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.611554    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-home\") pod \"0f2c92c0-79d9-447c-98c6-d7902962fffa\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") "
Sep 07 13:28:45.611597 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.611586    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f2c92c0-79d9-447c-98c6-d7902962fffa-tls-certs\") pod \"0f2c92c0-79d9-447c-98c6-d7902962fffa\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") "
Sep 07 13:28:45.611826 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.611633    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-model-cache\") pod \"0f2c92c0-79d9-447c-98c6-d7902962fffa\" (UID: \"0f2c92c0-79d9-447c-98c6-d7902962fffa\") "
Sep 07 13:28:45.611826 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.611689    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f2c92c0-79d9-447c-98c6-d7902962fffa" (UID: "0f2c92c0-79d9-447c-98c6-d7902962fffa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:28:45.611949 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.611841    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-home" (OuterVolumeSpecName: "home") pod "0f2c92c0-79d9-447c-98c6-d7902962fffa" (UID: "0f2c92c0-79d9-447c-98c6-d7902962fffa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:28:45.611949 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.611886    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f2c92c0-79d9-447c-98c6-d7902962fffa" (UID: "0f2c92c0-79d9-447c-98c6-d7902962fffa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:28:45.612074 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.612060    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-home\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:28:45.612177 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.612077    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-model-cache\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:28:45.612177 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.612087    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-tmp-dir\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:28:45.613721 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.613697    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-dshm" (OuterVolumeSpecName: "dshm") pod "0f2c92c0-79d9-447c-98c6-d7902962fffa" (UID: "0f2c92c0-79d9-447c-98c6-d7902962fffa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:28:45.613817 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.613732    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2c92c0-79d9-447c-98c6-d7902962fffa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f2c92c0-79d9-447c-98c6-d7902962fffa" (UID: "0f2c92c0-79d9-447c-98c6-d7902962fffa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:28:45.613817 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.613744    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2c92c0-79d9-447c-98c6-d7902962fffa-kube-api-access-q5kxr" (OuterVolumeSpecName: "kube-api-access-q5kxr") pod "0f2c92c0-79d9-447c-98c6-d7902962fffa" (UID: "0f2c92c0-79d9-447c-98c6-d7902962fffa"). InnerVolumeSpecName "kube-api-access-q5kxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:28:45.712815 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.712777    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f2c92c0-79d9-447c-98c6-d7902962fffa-dshm\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:28:45.712815 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.712811    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5kxr\" (UniqueName: \"kubernetes.io/projected/0f2c92c0-79d9-447c-98c6-d7902962fffa-kube-api-access-q5kxr\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:28:45.712815 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:45.712823    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f2c92c0-79d9-447c-98c6-d7902962fffa-tls-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:28:46.015855 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.015741    2565 generic.go:356] "Generic (PLEG): container finished" podID="0f2c92c0-79d9-447c-98c6-d7902962fffa" containerID="4bb0dc417dbbd121322901b7a105e6834e0d900d63a702e0e6e07f3fd43f7659" exitCode=0
Sep 07 13:28:46.015855 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.015790    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"
Sep 07 13:28:46.015855 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.015814    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh" event={"ID":"0f2c92c0-79d9-447c-98c6-d7902962fffa","Type":"ContainerDied","Data":"4bb0dc417dbbd121322901b7a105e6834e0d900d63a702e0e6e07f3fd43f7659"}
Sep 07 13:28:46.015855 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.015855    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh" event={"ID":"0f2c92c0-79d9-447c-98c6-d7902962fffa","Type":"ContainerDied","Data":"109e88b9772bc6caa780ce404714247953762499f37abb74b73a9c4cac613f79"}
Sep 07 13:28:46.016225 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.015880    2565 scope.go:117] "RemoveContainer" containerID="4bb0dc417dbbd121322901b7a105e6834e0d900d63a702e0e6e07f3fd43f7659"
Sep 07 13:28:46.026171 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.026151    2565 scope.go:117] "RemoveContainer" containerID="4bb0dc417dbbd121322901b7a105e6834e0d900d63a702e0e6e07f3fd43f7659"
Sep 07 13:28:46.026490 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:28:46.026470    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb0dc417dbbd121322901b7a105e6834e0d900d63a702e0e6e07f3fd43f7659\": container with ID starting with 4bb0dc417dbbd121322901b7a105e6834e0d900d63a702e0e6e07f3fd43f7659 not found: ID does not exist" containerID="4bb0dc417dbbd121322901b7a105e6834e0d900d63a702e0e6e07f3fd43f7659"
Sep 07 13:28:46.026567 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.026499    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb0dc417dbbd121322901b7a105e6834e0d900d63a702e0e6e07f3fd43f7659"} err="failed to get container status \"4bb0dc417dbbd121322901b7a105e6834e0d900d63a702e0e6e07f3fd43f7659\": rpc error: code = NotFound desc = could not find container \"4bb0dc417dbbd121322901b7a105e6834e0d900d63a702e0e6e07f3fd43f7659\": container with ID starting with 4bb0dc417dbbd121322901b7a105e6834e0d900d63a702e0e6e07f3fd43f7659 not found: ID does not exist"
Sep 07 13:28:46.038706 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.038678    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"]
Sep 07 13:28:46.045601 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.045577    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-580612d9-kserve-86cf4fd595-ppdxh"]
Sep 07 13:28:46.252694 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.252644    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx" podUID="b256e91a-64d6-43bb-8af6-d561ef1d4cf4" containerName="main" containerID="cri-o://aa293c2c9a8060d59fb173af21555d6c0d221db53881a4ed59c17a6b510b0030" gracePeriod=45
Sep 07 13:28:46.502313 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.502283    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:28:46.617689 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.617589    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:28:46.621247 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.621216    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8mwn\" (UniqueName: \"kubernetes.io/projected/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-kube-api-access-n8mwn\") pod \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") "
Sep 07 13:28:46.621430 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.621252    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-dshm\") pod \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") "
Sep 07 13:28:46.621430 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.621295    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-home\") pod \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") "
Sep 07 13:28:46.621430 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.621340    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-tls-certs\") pod \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") "
Sep 07 13:28:46.621430 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.621393    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-tmp-dir\") pod \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") "
Sep 07 13:28:46.621665 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.621475    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-model-cache\") pod \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\" (UID: \"b256e91a-64d6-43bb-8af6-d561ef1d4cf4\") "
Sep 07 13:28:46.621665 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.621599    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-home" (OuterVolumeSpecName: "home") pod "b256e91a-64d6-43bb-8af6-d561ef1d4cf4" (UID: "b256e91a-64d6-43bb-8af6-d561ef1d4cf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:28:46.621787 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.621730    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b256e91a-64d6-43bb-8af6-d561ef1d4cf4" (UID: "b256e91a-64d6-43bb-8af6-d561ef1d4cf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:28:46.621787 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.621774    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-model-cache" (OuterVolumeSpecName: "model-cache") pod "b256e91a-64d6-43bb-8af6-d561ef1d4cf4" (UID: "b256e91a-64d6-43bb-8af6-d561ef1d4cf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:28:46.621889 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.621813    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-home\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:28:46.623690 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.623655    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-dshm" (OuterVolumeSpecName: "dshm") pod "b256e91a-64d6-43bb-8af6-d561ef1d4cf4" (UID: "b256e91a-64d6-43bb-8af6-d561ef1d4cf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 13:28:46.623813 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.623701    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-kube-api-access-n8mwn" (OuterVolumeSpecName: "kube-api-access-n8mwn") pod "b256e91a-64d6-43bb-8af6-d561ef1d4cf4" (UID: "b256e91a-64d6-43bb-8af6-d561ef1d4cf4"). InnerVolumeSpecName "kube-api-access-n8mwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:28:46.623813 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.623719    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b256e91a-64d6-43bb-8af6-d561ef1d4cf4" (UID: "b256e91a-64d6-43bb-8af6-d561ef1d4cf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:28:46.722665 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.722632    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-model-cache\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:28:46.722665 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.722662    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8mwn\" (UniqueName: \"kubernetes.io/projected/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-kube-api-access-n8mwn\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:28:46.722665 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.722671    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-dshm\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:28:46.722994 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.722680    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-tls-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:28:46.722994 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:46.722690    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b256e91a-64d6-43bb-8af6-d561ef1d4cf4-tmp-dir\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:28:47.023372 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:47.023273    2565 generic.go:356] "Generic (PLEG): container finished" podID="b256e91a-64d6-43bb-8af6-d561ef1d4cf4" containerID="aa293c2c9a8060d59fb173af21555d6c0d221db53881a4ed59c17a6b510b0030" exitCode=0
Sep 07 13:28:47.023372 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:47.023313    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx" event={"ID":"b256e91a-64d6-43bb-8af6-d561ef1d4cf4","Type":"ContainerDied","Data":"aa293c2c9a8060d59fb173af21555d6c0d221db53881a4ed59c17a6b510b0030"}
Sep 07 13:28:47.023372 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:47.023319    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"
Sep 07 13:28:47.023372 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:47.023363    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx" event={"ID":"b256e91a-64d6-43bb-8af6-d561ef1d4cf4","Type":"ContainerDied","Data":"01bb007e769de2686d52f9c35640e306811f106e070f7f020807f6b60ada1bbb"}
Sep 07 13:28:47.023653 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:47.023380    2565 scope.go:117] "RemoveContainer" containerID="aa293c2c9a8060d59fb173af21555d6c0d221db53881a4ed59c17a6b510b0030"
Sep 07 13:28:47.032860 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:47.032843    2565 scope.go:117] "RemoveContainer" containerID="aa293c2c9a8060d59fb173af21555d6c0d221db53881a4ed59c17a6b510b0030"
Sep 07 13:28:47.033136 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:28:47.033100    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa293c2c9a8060d59fb173af21555d6c0d221db53881a4ed59c17a6b510b0030\": container with ID starting with aa293c2c9a8060d59fb173af21555d6c0d221db53881a4ed59c17a6b510b0030 not found: ID does not exist" containerID="aa293c2c9a8060d59fb173af21555d6c0d221db53881a4ed59c17a6b510b0030"
Sep 07 13:28:47.033136 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:47.033126    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa293c2c9a8060d59fb173af21555d6c0d221db53881a4ed59c17a6b510b0030"} err="failed to get container status \"aa293c2c9a8060d59fb173af21555d6c0d221db53881a4ed59c17a6b510b0030\": rpc error: code = NotFound desc = could not find container \"aa293c2c9a8060d59fb173af21555d6c0d221db53881a4ed59c17a6b510b0030\": container with ID starting with aa293c2c9a8060d59fb173af21555d6c0d221db53881a4ed59c17a6b510b0030 not found: ID does not exist"
Sep 07 13:28:47.048584 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:47.048560    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"]
Sep 07 13:28:47.054827 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:47.054805    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-580612d9-kserve-584bf45b99-jmclx"]
Sep 07 13:28:47.489552 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:47.489519    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f2c92c0-79d9-447c-98c6-d7902962fffa" path="/var/lib/kubelet/pods/0f2c92c0-79d9-447c-98c6-d7902962fffa/volumes"
Sep 07 13:28:47.489933 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:47.489919    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b256e91a-64d6-43bb-8af6-d561ef1d4cf4" path="/var/lib/kubelet/pods/b256e91a-64d6-43bb-8af6-d561ef1d4cf4/volumes"
Sep 07 13:28:51.421487 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.421446    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"]
Sep 07 13:28:51.421886 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.421873    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2c92c0-79d9-447c-98c6-d7902962fffa" containerName="main"
Sep 07 13:28:51.421886 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.421887    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2c92c0-79d9-447c-98c6-d7902962fffa" containerName="main"
Sep 07 13:28:51.421977 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.421896    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b256e91a-64d6-43bb-8af6-d561ef1d4cf4" containerName="main"
Sep 07 13:28:51.421977 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.421902    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b256e91a-64d6-43bb-8af6-d561ef1d4cf4" containerName="main"
Sep 07 13:28:51.421977 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.421966    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b256e91a-64d6-43bb-8af6-d561ef1d4cf4" containerName="main"
Sep 07 13:28:51.421977 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.421976    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2c92c0-79d9-447c-98c6-d7902962fffa" containerName="main"
Sep 07 13:28:51.426457 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.426439    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:51.435021 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.434997    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 07 13:28:51.435139 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.434999    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 13:28:51.435139 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.435099    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 13:28:51.435907 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.435886    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kvznr\""
Sep 07 13:28:51.435995 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.435889    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 13:28:51.439379 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.439359    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"]
Sep 07 13:28:51.470616 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.470580    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9939e1e3-752e-44de-9b36-8ecddb0a500d-maas-api-tls\") pod \"maas-api-6d966df4d9-74vrv\" (UID: \"9939e1e3-752e-44de-9b36-8ecddb0a500d\") " pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:51.470798 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.470621    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9939e1e3-752e-44de-9b36-8ecddb0a500d-openshift-service-ca\") pod \"maas-api-6d966df4d9-74vrv\" (UID: \"9939e1e3-752e-44de-9b36-8ecddb0a500d\") " pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:51.470798 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.470651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb7dr\" (UniqueName: \"kubernetes.io/projected/9939e1e3-752e-44de-9b36-8ecddb0a500d-kube-api-access-gb7dr\") pod \"maas-api-6d966df4d9-74vrv\" (UID: \"9939e1e3-752e-44de-9b36-8ecddb0a500d\") " pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:51.470798 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.470734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9939e1e3-752e-44de-9b36-8ecddb0a500d-metrics-certs\") pod \"maas-api-6d966df4d9-74vrv\" (UID: \"9939e1e3-752e-44de-9b36-8ecddb0a500d\") " pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:51.571606 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.571566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9939e1e3-752e-44de-9b36-8ecddb0a500d-maas-api-tls\") pod \"maas-api-6d966df4d9-74vrv\" (UID: \"9939e1e3-752e-44de-9b36-8ecddb0a500d\") " pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:51.571606 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.571608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9939e1e3-752e-44de-9b36-8ecddb0a500d-openshift-service-ca\") pod \"maas-api-6d966df4d9-74vrv\" (UID: \"9939e1e3-752e-44de-9b36-8ecddb0a500d\") " pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:51.571881 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.571628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb7dr\" (UniqueName: \"kubernetes.io/projected/9939e1e3-752e-44de-9b36-8ecddb0a500d-kube-api-access-gb7dr\") pod \"maas-api-6d966df4d9-74vrv\" (UID: \"9939e1e3-752e-44de-9b36-8ecddb0a500d\") " pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:51.571881 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.571761    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9939e1e3-752e-44de-9b36-8ecddb0a500d-metrics-certs\") pod \"maas-api-6d966df4d9-74vrv\" (UID: \"9939e1e3-752e-44de-9b36-8ecddb0a500d\") " pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:51.572310 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.572286    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9939e1e3-752e-44de-9b36-8ecddb0a500d-openshift-service-ca\") pod \"maas-api-6d966df4d9-74vrv\" (UID: \"9939e1e3-752e-44de-9b36-8ecddb0a500d\") " pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:51.574240 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.574219    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9939e1e3-752e-44de-9b36-8ecddb0a500d-metrics-certs\") pod \"maas-api-6d966df4d9-74vrv\" (UID: \"9939e1e3-752e-44de-9b36-8ecddb0a500d\") " pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:51.574339 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.574297    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9939e1e3-752e-44de-9b36-8ecddb0a500d-maas-api-tls\") pod \"maas-api-6d966df4d9-74vrv\" (UID: \"9939e1e3-752e-44de-9b36-8ecddb0a500d\") " pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:51.582308 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.582285    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb7dr\" (UniqueName: \"kubernetes.io/projected/9939e1e3-752e-44de-9b36-8ecddb0a500d-kube-api-access-gb7dr\") pod \"maas-api-6d966df4d9-74vrv\" (UID: \"9939e1e3-752e-44de-9b36-8ecddb0a500d\") " pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:51.736656 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:51.736564    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:52.019749 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:52.019671    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"]
Sep 07 13:28:52.022276 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:28:52.022249    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9939e1e3_752e_44de_9b36_8ecddb0a500d.slice/crio-0b8bb3463e600bd7e6fb516238ff6bd5a4b32e6d9c46b754d10b09a97b11a233 WatchSource:0}: Error finding container 0b8bb3463e600bd7e6fb516238ff6bd5a4b32e6d9c46b754d10b09a97b11a233: Status 404 returned error can't find the container with id 0b8bb3463e600bd7e6fb516238ff6bd5a4b32e6d9c46b754d10b09a97b11a233
Sep 07 13:28:52.024017 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:52.024001    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 13:28:52.052733 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:52.052698    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv" event={"ID":"9939e1e3-752e-44de-9b36-8ecddb0a500d","Type":"ContainerStarted","Data":"0b8bb3463e600bd7e6fb516238ff6bd5a4b32e6d9c46b754d10b09a97b11a233"}
Sep 07 13:28:52.815147 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:52.815041    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:28:55.074101 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:55.074018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv" event={"ID":"9939e1e3-752e-44de-9b36-8ecddb0a500d","Type":"ContainerStarted","Data":"0cdeb35027b1c76f23ec0fa2afb20c4c7833b5feb00743d5bf113f9cb4a3b97d"}
Sep 07 13:28:55.074641 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:55.074145    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:55.096994 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:55.096946    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv" podStartSLOduration=1.744513969 podStartE2EDuration="4.096930692s" totalImagesPullingTime="2.352416723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:28:55.095753506 +0000 UTC m=+2152.126619831" watchObservedRunningTime="2026-09-07 13:28:55.096930692 +0000 UTC m=+2152.127797002"
Sep 07 13:28:59.007852 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.007813    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:28:59.405145 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.405103    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"]
Sep 07 13:28:59.405356 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.405282    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv" podUID="9939e1e3-752e-44de-9b36-8ecddb0a500d" containerName="maas-api" containerID="cri-o://0cdeb35027b1c76f23ec0fa2afb20c4c7833b5feb00743d5bf113f9cb4a3b97d" gracePeriod=30
Sep 07 13:28:59.410384 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.410361    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:59.656772 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.656712    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:28:59.754704 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.754667    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb7dr\" (UniqueName: \"kubernetes.io/projected/9939e1e3-752e-44de-9b36-8ecddb0a500d-kube-api-access-gb7dr\") pod \"9939e1e3-752e-44de-9b36-8ecddb0a500d\" (UID: \"9939e1e3-752e-44de-9b36-8ecddb0a500d\") "
Sep 07 13:28:59.754878 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.754741    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9939e1e3-752e-44de-9b36-8ecddb0a500d-maas-api-tls\") pod \"9939e1e3-752e-44de-9b36-8ecddb0a500d\" (UID: \"9939e1e3-752e-44de-9b36-8ecddb0a500d\") "
Sep 07 13:28:59.754878 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.754772    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9939e1e3-752e-44de-9b36-8ecddb0a500d-metrics-certs\") pod \"9939e1e3-752e-44de-9b36-8ecddb0a500d\" (UID: \"9939e1e3-752e-44de-9b36-8ecddb0a500d\") "
Sep 07 13:28:59.755014 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.754886    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9939e1e3-752e-44de-9b36-8ecddb0a500d-openshift-service-ca\") pod \"9939e1e3-752e-44de-9b36-8ecddb0a500d\" (UID: \"9939e1e3-752e-44de-9b36-8ecddb0a500d\") "
Sep 07 13:28:59.755367 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.755306    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9939e1e3-752e-44de-9b36-8ecddb0a500d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9939e1e3-752e-44de-9b36-8ecddb0a500d" (UID: "9939e1e3-752e-44de-9b36-8ecddb0a500d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 13:28:59.757227 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.757196    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9939e1e3-752e-44de-9b36-8ecddb0a500d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9939e1e3-752e-44de-9b36-8ecddb0a500d" (UID: "9939e1e3-752e-44de-9b36-8ecddb0a500d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:28:59.757227 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.757210    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9939e1e3-752e-44de-9b36-8ecddb0a500d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9939e1e3-752e-44de-9b36-8ecddb0a500d" (UID: "9939e1e3-752e-44de-9b36-8ecddb0a500d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 13:28:59.757400 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.757213    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9939e1e3-752e-44de-9b36-8ecddb0a500d-kube-api-access-gb7dr" (OuterVolumeSpecName: "kube-api-access-gb7dr") pod "9939e1e3-752e-44de-9b36-8ecddb0a500d" (UID: "9939e1e3-752e-44de-9b36-8ecddb0a500d"). InnerVolumeSpecName "kube-api-access-gb7dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 13:28:59.856579 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.856535    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb7dr\" (UniqueName: \"kubernetes.io/projected/9939e1e3-752e-44de-9b36-8ecddb0a500d-kube-api-access-gb7dr\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:28:59.856579 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.856571    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9939e1e3-752e-44de-9b36-8ecddb0a500d-maas-api-tls\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:28:59.856579 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.856583    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9939e1e3-752e-44de-9b36-8ecddb0a500d-metrics-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:28:59.856812 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:28:59.856593    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9939e1e3-752e-44de-9b36-8ecddb0a500d-openshift-service-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\""
Sep 07 13:29:00.105978 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:00.105940    2565 generic.go:356] "Generic (PLEG): container finished" podID="9939e1e3-752e-44de-9b36-8ecddb0a500d" containerID="0cdeb35027b1c76f23ec0fa2afb20c4c7833b5feb00743d5bf113f9cb4a3b97d" exitCode=0
Sep 07 13:29:00.106442 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:00.106000    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"
Sep 07 13:29:00.106442 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:00.106040    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv" event={"ID":"9939e1e3-752e-44de-9b36-8ecddb0a500d","Type":"ContainerDied","Data":"0cdeb35027b1c76f23ec0fa2afb20c4c7833b5feb00743d5bf113f9cb4a3b97d"}
Sep 07 13:29:00.106442 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:00.106114    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv" event={"ID":"9939e1e3-752e-44de-9b36-8ecddb0a500d","Type":"ContainerDied","Data":"0b8bb3463e600bd7e6fb516238ff6bd5a4b32e6d9c46b754d10b09a97b11a233"}
Sep 07 13:29:00.106442 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:00.106148    2565 scope.go:117] "RemoveContainer" containerID="0cdeb35027b1c76f23ec0fa2afb20c4c7833b5feb00743d5bf113f9cb4a3b97d"
Sep 07 13:29:00.116098 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:00.116067    2565 scope.go:117] "RemoveContainer" containerID="0cdeb35027b1c76f23ec0fa2afb20c4c7833b5feb00743d5bf113f9cb4a3b97d"
Sep 07 13:29:00.116407 ip-10-0-141-58 kubenswrapper[2565]: E0907 13:29:00.116379    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cdeb35027b1c76f23ec0fa2afb20c4c7833b5feb00743d5bf113f9cb4a3b97d\": container with ID starting with 0cdeb35027b1c76f23ec0fa2afb20c4c7833b5feb00743d5bf113f9cb4a3b97d not found: ID does not exist" containerID="0cdeb35027b1c76f23ec0fa2afb20c4c7833b5feb00743d5bf113f9cb4a3b97d"
Sep 07 13:29:00.116574 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:00.116546    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cdeb35027b1c76f23ec0fa2afb20c4c7833b5feb00743d5bf113f9cb4a3b97d"} err="failed to get container status \"0cdeb35027b1c76f23ec0fa2afb20c4c7833b5feb00743d5bf113f9cb4a3b97d\": rpc error: code = NotFound desc = could not find container \"0cdeb35027b1c76f23ec0fa2afb20c4c7833b5feb00743d5bf113f9cb4a3b97d\": container with ID starting with 0cdeb35027b1c76f23ec0fa2afb20c4c7833b5feb00743d5bf113f9cb4a3b97d not found: ID does not exist"
Sep 07 13:29:00.133564 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:00.133537    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"]
Sep 07 13:29:00.139032 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:00.138995    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d966df4d9-74vrv"]
Sep 07 13:29:01.487707 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:01.487673    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9939e1e3-752e-44de-9b36-8ecddb0a500d" path="/var/lib/kubelet/pods/9939e1e3-752e-44de-9b36-8ecddb0a500d/volumes"
Sep 07 13:29:03.922463 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:03.922430    2565 scope.go:117] "RemoveContainer" containerID="dee3659bd996f38dd76e53158119ffdf95f51c2367b2f482f522fde003bb0ed3"
Sep 07 13:29:03.930914 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:03.930886    2565 scope.go:117] "RemoveContainer" containerID="984cbc952005eefc4eda821512ff93939d6841ecb89f91cf7298221b970436a7"
Sep 07 13:29:03.938894 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:03.938875    2565 scope.go:117] "RemoveContainer" containerID="c988191ec85c7fc4e66cbcbb922f27d5b746bd40bdcad62d0af581daa2664e21"
Sep 07 13:29:10.313878 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:10.313833    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:29:17.706313 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:17.706277    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:29:22.709023 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:22.708990    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-z2nx8"]
Sep 07 13:29:31.992119 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:31.992088    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cc4b9478f-md9pt_182dd5d3-d39d-4aa1-9708-a912ad59a5f7/copy-manifests/0.log"
Sep 07 13:29:32.011699 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:32.011669    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cc4b9478f-md9pt_182dd5d3-d39d-4aa1-9708-a912ad59a5f7/manager/0.log"
Sep 07 13:29:34.308742 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:34.308704    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-z2nx8_eba692f7-75ab-487a-94c0-6e90e89ea54c/limitador/0.log"
Sep 07 13:29:35.144124 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:35.144092    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-558c85b7d6-q7pcz_354de67c-b720-402c-9307-6d8467a103b2/kube-auth-proxy/0.log"
Sep 07 13:29:35.269525 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:35.269494    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-pstv4_08fad632-9c22-4303-8c09-aa25f954ccaa/istio-proxy/1.log"
Sep 07 13:29:35.397051 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:35.396968    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-gqz94_338d8b09-0346-4935-8010-32425a4c3de7/payload-pre-processing/0.log"
Sep 07 13:29:35.975659 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:35.975626    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-bzxc7_41fcd958-faa4-4ed0-8d12-8ee3c19f2d22/main/0.log"
Sep 07 13:29:36.544908 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:36.544866    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jdbxn_2bf15fcf-9d6d-4cba-b1d7-9e025c3ca11f/main/0.log"
Sep 07 13:29:36.661001 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:36.660972    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-fsqls_1781c947-2aaf-43b7-9773-5daac436de70/main/0.log"
Sep 07 13:29:45.732780 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:45.732743    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vjgrh_b22d72c4-fb0e-4f78-9e7e-0550e6928616/global-pull-secret-syncer/0.log"
Sep 07 13:29:45.834556 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:45.834523    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-w7hq9_da9073e8-463b-4ec5-b1e2-03a9260d2471/konnectivity-agent/0.log"
Sep 07 13:29:45.978441 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:45.978401    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-58.ec2.internal_899f0bbad87a70907b72c03ee3b7d029/haproxy/0.log"
Sep 07 13:29:50.527190 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:50.527154    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-z2nx8_eba692f7-75ab-487a-94c0-6e90e89ea54c/limitador/0.log"
Sep 07 13:29:52.176887 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.176842    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f785a771-6a28-441e-a35a-bb427caaf8c7/alertmanager/0.log"
Sep 07 13:29:52.202110 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.202087    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f785a771-6a28-441e-a35a-bb427caaf8c7/config-reloader/0.log"
Sep 07 13:29:52.228496 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.228469    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f785a771-6a28-441e-a35a-bb427caaf8c7/kube-rbac-proxy-web/0.log"
Sep 07 13:29:52.256154 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.256109    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f785a771-6a28-441e-a35a-bb427caaf8c7/kube-rbac-proxy/0.log"
Sep 07 13:29:52.275624 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.275585    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f785a771-6a28-441e-a35a-bb427caaf8c7/kube-rbac-proxy-metric/0.log"
Sep 07 13:29:52.299320 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.299285    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f785a771-6a28-441e-a35a-bb427caaf8c7/prom-label-proxy/0.log"
Sep 07 13:29:52.329530 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.329502    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f785a771-6a28-441e-a35a-bb427caaf8c7/init-config-reloader/0.log"
Sep 07 13:29:52.417157 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.417127    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-q7hxn_090dc0b6-5c88-4c1a-9138-6011e61be2cd/kube-state-metrics/0.log"
Sep 07 13:29:52.452785 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.452702    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-q7hxn_090dc0b6-5c88-4c1a-9138-6011e61be2cd/kube-rbac-proxy-main/0.log"
Sep 07 13:29:52.489897 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.489873    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-q7hxn_090dc0b6-5c88-4c1a-9138-6011e61be2cd/kube-rbac-proxy-self/0.log"
Sep 07 13:29:52.527378 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.527312    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-567854bfc5-m9hbl_9ef7c14b-87f2-4057-aee8-b32c36d0535a/metrics-server/0.log"
Sep 07 13:29:52.598072 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.598040    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-5sp6l_c32174db-fc07-4988-9eb2-19b2908afa4b/monitoring-plugin/0.log"
Sep 07 13:29:52.640086 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.640054    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-279tc_9a852057-5609-46ab-b910-af4397e8165a/node-exporter/0.log"
Sep 07 13:29:52.660289 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.660253    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-279tc_9a852057-5609-46ab-b910-af4397e8165a/kube-rbac-proxy/0.log"
Sep 07 13:29:52.682605 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.682586    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-279tc_9a852057-5609-46ab-b910-af4397e8165a/init-textfile/0.log"
Sep 07 13:29:52.946285 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.946252    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-lznzr_64a0a42b-c68c-4645-98af-30682c2b4a35/kube-rbac-proxy-main/0.log"
Sep 07 13:29:52.972800 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.972775    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-lznzr_64a0a42b-c68c-4645-98af-30682c2b4a35/kube-rbac-proxy-self/0.log"
Sep 07 13:29:52.994871 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:52.994843    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-lznzr_64a0a42b-c68c-4645-98af-30682c2b4a35/openshift-state-metrics/0.log"
Sep 07 13:29:53.038854 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.038829    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_539502f5-568f-4143-a9b5-e2a926f85a08/prometheus/0.log"
Sep 07 13:29:53.075996 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.075966    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_539502f5-568f-4143-a9b5-e2a926f85a08/config-reloader/0.log"
Sep 07 13:29:53.140191 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.140155    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_539502f5-568f-4143-a9b5-e2a926f85a08/thanos-sidecar/0.log"
Sep 07 13:29:53.212496 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.212468    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_539502f5-568f-4143-a9b5-e2a926f85a08/kube-rbac-proxy-web/0.log"
Sep 07 13:29:53.278654 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.278566    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_539502f5-568f-4143-a9b5-e2a926f85a08/kube-rbac-proxy/0.log"
Sep 07 13:29:53.304744 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.304711    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_539502f5-568f-4143-a9b5-e2a926f85a08/kube-rbac-proxy-thanos/0.log"
Sep 07 13:29:53.335341 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.335287    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_539502f5-568f-4143-a9b5-e2a926f85a08/init-config-reloader/0.log"
Sep 07 13:29:53.362610 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.362580    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-z45nq_8df4290a-39f3-4ea5-b30e-6ccd32e57bca/prometheus-operator/0.log"
Sep 07 13:29:53.381387 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.381358    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-z45nq_8df4290a-39f3-4ea5-b30e-6ccd32e57bca/kube-rbac-proxy/0.log"
Sep 07 13:29:53.410963 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.410933    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-js568_13de3759-1c38-4740-8eff-496554cb5525/prometheus-operator-admission-webhook/0.log"
Sep 07 13:29:53.442715 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.442682    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8c5dd6dc4-mlb66_e62da2fb-574e-48fd-b94b-ea0e50807ce0/telemeter-client/0.log"
Sep 07 13:29:53.465660 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.465631    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8c5dd6dc4-mlb66_e62da2fb-574e-48fd-b94b-ea0e50807ce0/reload/0.log"
Sep 07 13:29:53.495739 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.495706    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8c5dd6dc4-mlb66_e62da2fb-574e-48fd-b94b-ea0e50807ce0/kube-rbac-proxy/0.log"
Sep 07 13:29:53.544684 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.544660    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5874f5666d-jdf7t_fc908e22-0fe5-4665-abae-f5e5197585d7/thanos-query/0.log"
Sep 07 13:29:53.571854 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.571829    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5874f5666d-jdf7t_fc908e22-0fe5-4665-abae-f5e5197585d7/kube-rbac-proxy-web/0.log"
Sep 07 13:29:53.594872 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.594852    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5874f5666d-jdf7t_fc908e22-0fe5-4665-abae-f5e5197585d7/kube-rbac-proxy/0.log"
Sep 07 13:29:53.620578 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.620552    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5874f5666d-jdf7t_fc908e22-0fe5-4665-abae-f5e5197585d7/prom-label-proxy/0.log"
Sep 07 13:29:53.642720 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.642696    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5874f5666d-jdf7t_fc908e22-0fe5-4665-abae-f5e5197585d7/kube-rbac-proxy-rules/0.log"
Sep 07 13:29:53.684574 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:53.684521    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5874f5666d-jdf7t_fc908e22-0fe5-4665-abae-f5e5197585d7/kube-rbac-proxy-metrics/0.log"
Sep 07 13:29:55.680774 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.680738    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"]
Sep 07 13:29:55.681155 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.681129    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9939e1e3-752e-44de-9b36-8ecddb0a500d" containerName="maas-api"
Sep 07 13:29:55.681155 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.681141    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9939e1e3-752e-44de-9b36-8ecddb0a500d" containerName="maas-api"
Sep 07 13:29:55.681228 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.681213    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9939e1e3-752e-44de-9b36-8ecddb0a500d" containerName="maas-api"
Sep 07 13:29:55.684667 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.684642    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:55.688267 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.688245    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9djjs\"/\"default-dockercfg-zn4q2\""
Sep 07 13:29:55.688416 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.688394    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9djjs\"/\"openshift-service-ca.crt\""
Sep 07 13:29:55.688667 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.688647    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3576f73-6452-4b36-b813-19c183b5f951-lib-modules\") pod \"perf-node-gather-daemonset-zdw57\" (UID: \"d3576f73-6452-4b36-b813-19c183b5f951\") " pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:55.688734 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.688676    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3576f73-6452-4b36-b813-19c183b5f951-sys\") pod \"perf-node-gather-daemonset-zdw57\" (UID: \"d3576f73-6452-4b36-b813-19c183b5f951\") " pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:55.688734 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.688697    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d3576f73-6452-4b36-b813-19c183b5f951-proc\") pod \"perf-node-gather-daemonset-zdw57\" (UID: \"d3576f73-6452-4b36-b813-19c183b5f951\") " pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:55.688818 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.688780    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kxvn\" (UniqueName: \"kubernetes.io/projected/d3576f73-6452-4b36-b813-19c183b5f951-kube-api-access-5kxvn\") pod \"perf-node-gather-daemonset-zdw57\" (UID: \"d3576f73-6452-4b36-b813-19c183b5f951\") " pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:55.688877 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.688863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d3576f73-6452-4b36-b813-19c183b5f951-podres\") pod \"perf-node-gather-daemonset-zdw57\" (UID: \"d3576f73-6452-4b36-b813-19c183b5f951\") " pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:55.689015 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.688998    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9djjs\"/\"kube-root-ca.crt\""
Sep 07 13:29:55.697004 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.696978    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"]
Sep 07 13:29:55.790102 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.790061    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kxvn\" (UniqueName: \"kubernetes.io/projected/d3576f73-6452-4b36-b813-19c183b5f951-kube-api-access-5kxvn\") pod \"perf-node-gather-daemonset-zdw57\" (UID: \"d3576f73-6452-4b36-b813-19c183b5f951\") " pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:55.790304 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.790139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d3576f73-6452-4b36-b813-19c183b5f951-podres\") pod \"perf-node-gather-daemonset-zdw57\" (UID: \"d3576f73-6452-4b36-b813-19c183b5f951\") " pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:55.790304 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.790167    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3576f73-6452-4b36-b813-19c183b5f951-lib-modules\") pod \"perf-node-gather-daemonset-zdw57\" (UID: \"d3576f73-6452-4b36-b813-19c183b5f951\") " pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:55.790304 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.790185    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3576f73-6452-4b36-b813-19c183b5f951-sys\") pod \"perf-node-gather-daemonset-zdw57\" (UID: \"d3576f73-6452-4b36-b813-19c183b5f951\") " pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:55.790304 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.790203    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d3576f73-6452-4b36-b813-19c183b5f951-proc\") pod \"perf-node-gather-daemonset-zdw57\" (UID: \"d3576f73-6452-4b36-b813-19c183b5f951\") " pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:55.790304 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.790273    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d3576f73-6452-4b36-b813-19c183b5f951-proc\") pod \"perf-node-gather-daemonset-zdw57\" (UID: \"d3576f73-6452-4b36-b813-19c183b5f951\") " pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:55.790304 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.790286    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3576f73-6452-4b36-b813-19c183b5f951-sys\") pod \"perf-node-gather-daemonset-zdw57\" (UID: \"d3576f73-6452-4b36-b813-19c183b5f951\") " pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:55.790584 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.790318    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d3576f73-6452-4b36-b813-19c183b5f951-podres\") pod \"perf-node-gather-daemonset-zdw57\" (UID: \"d3576f73-6452-4b36-b813-19c183b5f951\") " pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:55.790584 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.790375    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3576f73-6452-4b36-b813-19c183b5f951-lib-modules\") pod \"perf-node-gather-daemonset-zdw57\" (UID: \"d3576f73-6452-4b36-b813-19c183b5f951\") " pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:55.799999 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.799973    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kxvn\" (UniqueName: \"kubernetes.io/projected/d3576f73-6452-4b36-b813-19c183b5f951-kube-api-access-5kxvn\") pod \"perf-node-gather-daemonset-zdw57\" (UID: \"d3576f73-6452-4b36-b813-19c183b5f951\") " pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:55.995347 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:55.995231    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:56.267674 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:56.267644    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"]
Sep 07 13:29:56.269950 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:29:56.269914    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd3576f73_6452_4b36_b813_19c183b5f951.slice/crio-f1effdbc8a71899a5f5e7d0417cf8fe8d72ae3b30c1fa4be3de4ab1279b4938d WatchSource:0}: Error finding container f1effdbc8a71899a5f5e7d0417cf8fe8d72ae3b30c1fa4be3de4ab1279b4938d: Status 404 returned error can't find the container with id f1effdbc8a71899a5f5e7d0417cf8fe8d72ae3b30c1fa4be3de4ab1279b4938d
Sep 07 13:29:56.402068 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:56.402039    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57" event={"ID":"d3576f73-6452-4b36-b813-19c183b5f951","Type":"ContainerStarted","Data":"f1effdbc8a71899a5f5e7d0417cf8fe8d72ae3b30c1fa4be3de4ab1279b4938d"}
Sep 07 13:29:57.407855 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:57.407817    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57" event={"ID":"d3576f73-6452-4b36-b813-19c183b5f951","Type":"ContainerStarted","Data":"e69087d30f0fdca166130f08e13e06fb445573266566012169ef19a4b8716220"}
Sep 07 13:29:57.408319 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:57.407963    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:29:57.433831 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:57.433775    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57" podStartSLOduration=2.433758465 podStartE2EDuration="2.433758465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 13:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:29:57.432174724 +0000 UTC m=+2214.463041059" watchObservedRunningTime="2026-09-07 13:29:57.433758465 +0000 UTC m=+2214.464624775"
Sep 07 13:29:58.510932 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:58.510902    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sb8hw_22574e81-953a-49b3-8305-6456988043d0/dns/0.log"
Sep 07 13:29:58.537084 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:58.537051    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sb8hw_22574e81-953a-49b3-8305-6456988043d0/kube-rbac-proxy/0.log"
Sep 07 13:29:58.622135 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:58.622105    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dqbpb_aaac4f3c-dfa1-4c7f-8e67-2e01a910c117/dns-node-resolver/0.log"
Sep 07 13:29:59.258474 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:29:59.258440    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7gz9f_d4450013-6c87-4e72-8722-20add36bb64a/node-ca/0.log"
Sep 07 13:30:00.149102 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:00.149059    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813130-c9zjh"]
Sep 07 13:30:00.153504 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:00.153483    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813130-c9zjh"
Sep 07 13:30:00.156079 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:00.156055    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flngs\""
Sep 07 13:30:00.162248 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:00.162210    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813130-c9zjh"]
Sep 07 13:30:00.164205 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:00.164185    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813130-c9zjh"
Sep 07 13:30:00.445022 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:00.444997    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813130-c9zjh"]
Sep 07 13:30:00.447006 ip-10-0-141-58 kubenswrapper[2565]: W0907 13:30:00.446973    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b18584_766c_46fc_a942_136bbb8efff1.slice/crio-fb69315afdc4843f1f3b3b5089abe89ed0bff80ce7953fc97b505741a7c61536 WatchSource:0}: Error finding container fb69315afdc4843f1f3b3b5089abe89ed0bff80ce7953fc97b505741a7c61536: Status 404 returned error can't find the container with id fb69315afdc4843f1f3b3b5089abe89ed0bff80ce7953fc97b505741a7c61536
Sep 07 13:30:00.889190 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:00.889149    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-558c85b7d6-q7pcz_354de67c-b720-402c-9307-6d8467a103b2/kube-auth-proxy/0.log"
Sep 07 13:30:00.923610 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:00.923571    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-pstv4_08fad632-9c22-4303-8c09-aa25f954ccaa/istio-proxy/0.log"
Sep 07 13:30:00.941764 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:00.941729    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-pstv4_08fad632-9c22-4303-8c09-aa25f954ccaa/istio-proxy/1.log"
Sep 07 13:30:00.987212 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:00.987124    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-gqz94_338d8b09-0346-4935-8010-32425a4c3de7/payload-pre-processing/0.log"
Sep 07 13:30:01.433458 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:01.433420    2565 generic.go:356] "Generic (PLEG): container finished" podID="d2b18584-766c-46fc-a942-136bbb8efff1" containerID="83c779b4d91f26ddf027537766096349fdcbbb642e4b0ba7a60e8481700d628a" exitCode=0
Sep 07 13:30:01.433794 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:01.433507    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813130-c9zjh" event={"ID":"d2b18584-766c-46fc-a942-136bbb8efff1","Type":"ContainerDied","Data":"83c779b4d91f26ddf027537766096349fdcbbb642e4b0ba7a60e8481700d628a"}
Sep 07 13:30:01.433794 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:01.433558    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813130-c9zjh" event={"ID":"d2b18584-766c-46fc-a942-136bbb8efff1","Type":"ContainerStarted","Data":"fb69315afdc4843f1f3b3b5089abe89ed0bff80ce7953fc97b505741a7c61536"}
Sep 07 13:30:02.085444 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:02.085406    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-brzmx_9a25b578-db78-49e9-b2d2-fb9440eef833/serve-healthcheck-canary/0.log"
Sep 07 13:30:02.566729 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:02.566706    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813130-c9zjh"
Sep 07 13:30:02.716841 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:02.716758    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-822q8_08e62ed2-77a2-4656-afad-a44e252f2ede/kube-rbac-proxy/0.log"
Sep 07 13:30:02.741727 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:02.741681    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-822q8_08e62ed2-77a2-4656-afad-a44e252f2ede/exporter/0.log"
Sep 07 13:30:02.764914 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:02.764878    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-822q8_08e62ed2-77a2-4656-afad-a44e252f2ede/extractor/0.log"
Sep 07 13:30:03.423298 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:03.423273    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9djjs/perf-node-gather-daemonset-zdw57"
Sep 07 13:30:03.445500 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:03.445466    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813130-c9zjh" event={"ID":"d2b18584-766c-46fc-a942-136bbb8efff1","Type":"ContainerDied","Data":"fb69315afdc4843f1f3b3b5089abe89ed0bff80ce7953fc97b505741a7c61536"}
Sep 07 13:30:03.445500 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:03.445495    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813130-c9zjh"
Sep 07 13:30:03.445709 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:03.445523    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb69315afdc4843f1f3b3b5089abe89ed0bff80ce7953fc97b505741a7c61536"
Sep 07 13:30:06.116770 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:06.116732    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cc4b9478f-md9pt_182dd5d3-d39d-4aa1-9708-a912ad59a5f7/manager/0.log"
Sep 07 13:30:06.135118 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:06.135084    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cc4b9478f-md9pt_182dd5d3-d39d-4aa1-9708-a912ad59a5f7/copy-manifests/0.log"
Sep 07 13:30:07.043672 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:07.043639    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-558c85b7d6-q7pcz_354de67c-b720-402c-9307-6d8467a103b2/kube-auth-proxy/0.log"
Sep 07 13:30:07.069352 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:07.069295    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-pstv4_08fad632-9c22-4303-8c09-aa25f954ccaa/istio-proxy/0.log"
Sep 07 13:30:07.087534 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:07.087506    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-pstv4_08fad632-9c22-4303-8c09-aa25f954ccaa/istio-proxy/1.log"
Sep 07 13:30:07.131841 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:07.131814    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-gqz94_338d8b09-0346-4935-8010-32425a4c3de7/payload-pre-processing/0.log"
Sep 07 13:30:07.487903 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:07.487868    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-78895d9f88-lwjfq_f0662053-a65e-4df5-b54d-c98d9d968cff/manager/0.log"
Sep 07 13:30:09.353079 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:09.353046    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-558c85b7d6-q7pcz_354de67c-b720-402c-9307-6d8467a103b2/kube-auth-proxy/0.log"
Sep 07 13:30:09.375774 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:09.375751    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-pstv4_08fad632-9c22-4303-8c09-aa25f954ccaa/istio-proxy/0.log"
Sep 07 13:30:09.395079 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:09.395050    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-pstv4_08fad632-9c22-4303-8c09-aa25f954ccaa/istio-proxy/1.log"
Sep 07 13:30:09.431964 ip-10-0-141-58 kubenswrapper[2565]: I0907 13:30:09.431933    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-gqz94_338d8b09-0346-4935-8010-32425a4c3de7/payload-pre-processing/0.log"